[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:48:58 dist-test-slave-t6c1 krb5kdc[4893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:48:58 dist-test-slave-t6c1 krb5kdc[4893](info): set up 2 sockets
Sep 29 02:48:58 dist-test-slave-t6c1 krb5kdc[4893](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 02:48:59.028725 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:48:59 dist-test-slave-t6c1 krb5kdc[4893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:48:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:48:59.211710 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:42331
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:42331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:48:59.523073 4906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:48:59.523386 4906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:48:59.523519 4906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:48:59.530861 4906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:48:59.531008 4906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:48:59.531080 4906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:48:59.531157 4906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:48:59.531214 4906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:48:59.541553 4906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:42331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:42331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:48:59.543052 4906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:48:59.544427 4906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:48:59.556118 4911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:48:59.556084 4914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:48:59.556049 4912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:48:59.556658 4906 server_base.cc:1034] running on GCE node
I20260929 02:48:59.557542 4906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:48:59.559110 4906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:48:59.560313 4906 hybrid_clock.cc:648] HybridClock initialized: now 1790650139560255 us; error 54 us; skew 500 ppm
I20260929 02:48:59.560659 4906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:48:59 dist-test-slave-t6c1 krb5kdc[4893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650139, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:48:59.567221 4906 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:48:59.568868 4906 webserver.cc:460] Webserver started at http://127.4.197.190:35749/ using document root <none> and password file <none>
I20260929 02:48:59.569577 4906 fs_manager.cc:359] Metadata directory not provided
I20260929 02:48:59.569694 4906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:48:59.569980 4906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:48:59.573381 4906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "06cca953e68e43649b3548ce26637a1f"
format_stamp: "Formatted at 2026-09-29 02:48:59 on dist-test-slave-t6c1"
server_key: "d9e281492596546275ba921569aac3f1"
server_key_iv: "dcddb5cf921bf15cf3076c76fb4f36aa"
server_key_version: "encryptionkey@0"
I20260929 02:48:59.574201 4906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "06cca953e68e43649b3548ce26637a1f"
format_stamp: "Formatted at 2026-09-29 02:48:59 on dist-test-slave-t6c1"
server_key: "d9e281492596546275ba921569aac3f1"
server_key_iv: "dcddb5cf921bf15cf3076c76fb4f36aa"
server_key_version: "encryptionkey@0"
I20260929 02:48:59.579696 4906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:48:59.583257 4921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:48:59.584659 4906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 02:48:59.584824 4906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "06cca953e68e43649b3548ce26637a1f"
format_stamp: "Formatted at 2026-09-29 02:48:59 on dist-test-slave-t6c1"
server_key: "d9e281492596546275ba921569aac3f1"
server_key_iv: "dcddb5cf921bf15cf3076c76fb4f36aa"
server_key_version: "encryptionkey@0"
I20260929 02:48:59.585004 4906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:48:59.605635 4906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:48:59.609817 4906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:48:59.610082 4906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:48:59.627602 4906 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:42331
I20260929 02:48:59.627647 4972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:42331 every 8 connection(s)
I20260929 02:48:59.629030 4906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:48:59.632861 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:48:59.637298 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 4906
I20260929 02:48:59.637627 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:48:59.640134 4886 external_mini_cluster.cc:1428] Setting key f3c8ab630fbc7e485f90b83f4380e9db
I20260929 02:48:59.647214 4973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f: Bootstrap starting.
I20260929 02:48:59.652156 4973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f: Neither blocks nor log segments found. Creating new log.
I20260929 02:48:59.653801 4973 log.cc:824] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:48:59.657998 4973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f: No bootstrap required, opened a new log
I20260929 02:48:59.670444 4973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cca953e68e43649b3548ce26637a1f" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42331 } }
I20260929 02:48:59.670859 4973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:48:59.671025 4973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06cca953e68e43649b3548ce26637a1f, State: Initialized, Role: FOLLOWER
I20260929 02:48:59.671821 4973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cca953e68e43649b3548ce26637a1f" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42331 } }
I20260929 02:48:59.672144 4973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:48:59.672286 4973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:48:59.672503 4973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:48:59.674791 4973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cca953e68e43649b3548ce26637a1f" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42331 } }
I20260929 02:48:59.675346 4973 leader_election.cc:302] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06cca953e68e43649b3548ce26637a1f; no voters:
I20260929 02:48:59.677809 4973 leader_election.cc:288] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:48:59.677908 4978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:48:59.680346 4973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 02:48:59 dist-test-slave-t6c1 krb5kdc[4893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:48:59.684772 4978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [term 1 LEADER]: Becoming Leader. State: Replica: 06cca953e68e43649b3548ce26637a1f, State: Running, Role: LEADER
I20260929 02:48:59.685516 4978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cca953e68e43649b3548ce26637a1f" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42331 } }
I20260929 02:48:59.688747 4979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06cca953e68e43649b3548ce26637a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cca953e68e43649b3548ce26637a1f" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42331 } } }
I20260929 02:48:59.689153 4979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [sys.catalog]: This master's current role is: LEADER
I20260929 02:48:59.695920 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:59.645326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.4.197.190:42331)
0929 02:48:59.649829 (+ 4503us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:59.649849 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:59.651844 (+ 1995us) server_negotiation.cc:407] Connection header received
0929 02:48:59.664996 (+ 13152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:59.665034 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:59.665701 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:59.666281 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:48:59.671247 (+ 4966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:59.673022 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:59.675845 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:59.677088 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:59.677308 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:59.685082 (+ 7774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:48:59.685096 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:48:59.685119 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:48:59.685182 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:48:59.689591 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:59.690798 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:59.690807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:59.690815 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:59.691083 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:59.691673 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:59.691679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:59.691683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:59.692398 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:48:59.692681 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:59.693960 (+ 1279us) server_negotiation.cc:299] Negotiation successful
0929 02:48:59.694429 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4242,"thread_start_us":3989,"threads_started":1}
I20260929 02:48:59.697932 4980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06cca953e68e43649b3548ce26637a1f. Latest consensus state: current_term: 1 leader_uuid: "06cca953e68e43649b3548ce26637a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cca953e68e43649b3548ce26637a1f" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42331 } } }
I20260929 02:48:59.698185 4985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:48:59.698367 4980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f [sys.catalog]: This master's current role is: LEADER
I20260929 02:48:59.703423 4985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:48:59.714525 4985 catalog_manager.cc:1347] Generated new cluster ID: 936d285056b64b75b77f773bfce793dd
I20260929 02:48:59.714704 4985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:48:59.734314 4985 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:48:59.736191 4985 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:48:59.745492 4985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06cca953e68e43649b3548ce26637a1f: Generated new TSK 0
I20260929 02:48:59.746325 4985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:48:59.790910 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42331
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:00.108408 5000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:00.108668 5000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:00.108759 5000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:00.115830 5000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:00.115929 5000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:00.116046 5000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:00.125252 5000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42331
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:00.126437 5000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:00.127740 5000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:00.140362 5008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:00.140478 5005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:00.140452 5006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:00.141041 5000 server_base.cc:1034] running on GCE node
I20260929 02:49:00.141681 5000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:00.142697 5000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:00.143900 5000 hybrid_clock.cc:648] HybridClock initialized: now 1790650140143855 us; error 60 us; skew 500 ppm
I20260929 02:49:00.144207 5000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:00 dist-test-slave-t6c1 krb5kdc[4893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650140, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:00.147296 5000 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:49:00.148779 5000 webserver.cc:460] Webserver started at http://127.4.197.129:41157/ using document root <none> and password file <none>
I20260929 02:49:00.149457 5000 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:00.149582 5000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:00.149878 5000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:00.152987 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "7f8f93b0dfbb48f19b2cf1cb55fd0b4d"
format_stamp: "Formatted at 2026-09-29 02:49:00 on dist-test-slave-t6c1"
server_key: "c24a29814caec6153ec81ab0d136f7d6"
server_key_iv: "873968f8814f53d8e8506bbb984840ce"
server_key_version: "encryptionkey@0"
I20260929 02:49:00.153718 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "7f8f93b0dfbb48f19b2cf1cb55fd0b4d"
format_stamp: "Formatted at 2026-09-29 02:49:00 on dist-test-slave-t6c1"
server_key: "c24a29814caec6153ec81ab0d136f7d6"
server_key_iv: "873968f8814f53d8e8506bbb984840ce"
server_key_version: "encryptionkey@0"
I20260929 02:49:00.158769 5000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:49:00.162196 5015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:00.163555 5000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:00.163700 5000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "7f8f93b0dfbb48f19b2cf1cb55fd0b4d"
format_stamp: "Formatted at 2026-09-29 02:49:00 on dist-test-slave-t6c1"
server_key: "c24a29814caec6153ec81ab0d136f7d6"
server_key_iv: "873968f8814f53d8e8506bbb984840ce"
server_key_version: "encryptionkey@0"
I20260929 02:49:00.163888 5000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:00.186349 5000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:00.190765 5000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:00.191066 5000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:00.192066 5000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:00.193415 5000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:00.193495 5000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:00.193622 5000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:00.193675 5000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:00.222738 5000 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:44435
I20260929 02:49:00.222783 5127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:44435 every 8 connection(s)
I20260929 02:49:00.224026 5000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:00.226213 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 5000
I20260929 02:49:00.226451 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:00.227245 4886 external_mini_cluster.cc:1428] Setting key e86003ab6684ec3f14e2309afb1cddfc
Sep 29 02:49:00 dist-test-slave-t6c1 krb5kdc[4893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650140, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
I20260929 02:49:00.244728 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:00.226592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:32923 (local address 127.4.197.190:42331)
0929 02:49:00.227047 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:00.227051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:00.228363 (+ 1312us) server_negotiation.cc:407] Connection header received
0929 02:49:00.230485 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:00.230491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:00.230569 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:00.230659 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:00.232262 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:00.232983 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:00.234535 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:00.234962 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:00.235072 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:00.238879 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:00.238885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:00.238888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:00.238953 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:00.241510 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:00.242265 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:00.242270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:00.242272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:00.242425 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:00.242814 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:00.242817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:00.242819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:00.243184 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:00.243295 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:00.244047 (+ 752us) server_negotiation.cc:299] Negotiation successful
0929 02:49:00.244290 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":214,"threads_started":1}
I20260929 02:49:00.245641 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:00.227365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42331 (local address 127.4.197.129:32923)
0929 02:49:00.228218 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:00.228261 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:00.230212 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:00.230942 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:00.230954 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:00.231501 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:00.232097 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:00.232115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:00.233139 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:00.233144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:00.234398 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:00.234410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:00.235177 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:00.235185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:00.235418 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:00.236058 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:00.236086 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:00.238606 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:00.241659 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:00.241667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:00.241683 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:00.242157 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:00.242528 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:00.242531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:00.242534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:00.242733 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:00.243306 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:00.243316 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:00.243682 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 02:49:00.243932 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 02:49:00.244366 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":304,"threads_started":1}
I20260929 02:49:00.247220 5128 heartbeater.cc:347] Connected to a master server at 127.4.197.190:42331
I20260929 02:49:00.247521 5128 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:00.248265 5128 heartbeater.cc:511] Master 127.4.197.190:42331 requested a full tablet report, sending...
I20260929 02:49:00.250347 4938 ts_manager.cc:194] Registered new tserver with Master: 7f8f93b0dfbb48f19b2cf1cb55fd0b4d (127.4.197.129:44435)
I20260929 02:49:00.251845 4938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:32923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:49:00.263550 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42331
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:00.575934 5134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:00.576182 5134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:00.576313 5134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:00.583310 5134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:00.583405 5134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:00.583551 5134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:00.592901 5134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42331
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:00.594071 5134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:00.595364 5134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:00.607833 5139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:00.607755 5142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:00.607784 5140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:00.608357 5134 server_base.cc:1034] running on GCE node
I20260929 02:49:00.609099 5134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:00.610114 5134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:00.611325 5134 hybrid_clock.cc:648] HybridClock initialized: now 1790650140611273 us; error 41 us; skew 500 ppm
I20260929 02:49:00.611655 5134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:00 dist-test-slave-t6c1 krb5kdc[4893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650140, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:00.614856 5134 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:49:00.616441 5134 webserver.cc:460] Webserver started at http://127.4.197.130:43493/ using document root <none> and password file <none>
I20260929 02:49:00.617152 5134 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:00.617296 5134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:00.617585 5134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:00.620513 5134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "cc5f3a05c25845c4a7415b4aad80d89a"
format_stamp: "Formatted at 2026-09-29 02:49:00 on dist-test-slave-t6c1"
server_key: "4c9057e6278f5424ecbf999d56195af6"
server_key_iv: "d90a922037588a2c1aff6975658c6800"
server_key_version: "encryptionkey@0"
I20260929 02:49:00.621253 5134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "cc5f3a05c25845c4a7415b4aad80d89a"
format_stamp: "Formatted at 2026-09-29 02:49:00 on dist-test-slave-t6c1"
server_key: "4c9057e6278f5424ecbf999d56195af6"
server_key_iv: "d90a922037588a2c1aff6975658c6800"
server_key_version: "encryptionkey@0"
I20260929 02:49:00.626483 5134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:49:00.630841 5149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:00.632510 5134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:00.632696 5134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "cc5f3a05c25845c4a7415b4aad80d89a"
format_stamp: "Formatted at 2026-09-29 02:49:00 on dist-test-slave-t6c1"
server_key: "4c9057e6278f5424ecbf999d56195af6"
server_key_iv: "d90a922037588a2c1aff6975658c6800"
server_key_version: "encryptionkey@0"
I20260929 02:49:00.632908 5134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:00.664135 5134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:00.668254 5134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:00.668519 5134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:00.669492 5134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:00.670841 5134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:00.670912 5134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:00.671083 5134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:00.671149 5134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:00.702064 5134 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:37327
I20260929 02:49:00.702116 5261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:37327 every 8 connection(s)
I20260929 02:49:00.703313 5134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:00.713111 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 5134
I20260929 02:49:00.713291 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:00.713685 4886 external_mini_cluster.cc:1428] Setting key 66ba7dcc0da57e0ec695b3b77c3370dc
Sep 29 02:49:00 dist-test-slave-t6c1 krb5kdc[4893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650140, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:00.722091 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:00.705647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:54725 (local address 127.4.197.190:42331)
0929 02:49:00.705839 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:00.705845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:00.707087 (+ 1242us) server_negotiation.cc:407] Connection header received
0929 02:49:00.708850 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:00.708856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:00.708932 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:00.709020 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:00.710430 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:00.711170 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:00.712638 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:00.713030 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:00.713148 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:00.716507 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:00.716513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:00.716518 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:00.716555 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:00.719406 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:00.720055 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:00.720059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:00.720061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:00.720193 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:00.720565 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:00.720570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:00.720573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:00.720816 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:00.720960 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:00.721545 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 02:49:00.721783 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:49:00.723034 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:00.706200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42331 (local address 127.4.197.130:54725)
0929 02:49:00.706872 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:00.706914 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:00.708590 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:00.709252 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:00.709260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:00.709726 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:00.710264 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:00.710281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:00.711311 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:00.711315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:00.712496 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:00.712512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:00.713237 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:00.713244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:00.713482 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:00.714109 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:00.714130 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:00.716298 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:00.719527 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:00.719532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:00.719543 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:00.719934 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:00.720292 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:00.720296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:00.720298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:00.720465 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:00.720936 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:00.720943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:00.721226 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 02:49:00.721447 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 02:49:00.721922 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":252,"threads_started":1}
I20260929 02:49:00.724412 5262 heartbeater.cc:347] Connected to a master server at 127.4.197.190:42331
I20260929 02:49:00.724678 5262 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:00.725348 5262 heartbeater.cc:511] Master 127.4.197.190:42331 requested a full tablet report, sending...
I20260929 02:49:00.726806 4938 ts_manager.cc:194] Registered new tserver with Master: cc5f3a05c25845c4a7415b4aad80d89a (127.4.197.130:37327)
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
I20260929 02:49:00.727982 4938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:54725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:49:00.748014 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42331
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:01.066507 5267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:01.066784 5267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:01.066941 5267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:01.074311 5267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:01.074429 5267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:01.074571 5267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:01.084652 5267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42331
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:01.086104 5267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:01.087664 5267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:01.102149 5272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:01.102205 5273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:01.102239 5275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:01.102974 5267 server_base.cc:1034] running on GCE node
I20260929 02:49:01.103981 5267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:01.104915 5267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:01.106117 5267 hybrid_clock.cc:648] HybridClock initialized: now 1790650141106070 us; error 35 us; skew 500 ppm
I20260929 02:49:01.106423 5267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[4893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650141, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:01.109498 5267 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:49:01.110901 5267 webserver.cc:460] Webserver started at http://127.4.197.131:39133/ using document root <none> and password file <none>
I20260929 02:49:01.111580 5267 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:01.111698 5267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:01.112010 5267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:01.114830 5267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "d57ab8ec45fd4c49956f589a82d25bf0"
format_stamp: "Formatted at 2026-09-29 02:49:01 on dist-test-slave-t6c1"
server_key: "1b2e038335391e74d3dae4e396712db8"
server_key_iv: "4c103dcddbb31ae843b774aaa81ad932"
server_key_version: "encryptionkey@0"
I20260929 02:49:01.115676 5267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "d57ab8ec45fd4c49956f589a82d25bf0"
format_stamp: "Formatted at 2026-09-29 02:49:01 on dist-test-slave-t6c1"
server_key: "1b2e038335391e74d3dae4e396712db8"
server_key_iv: "4c103dcddbb31ae843b774aaa81ad932"
server_key_version: "encryptionkey@0"
I20260929 02:49:01.120625 5267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:49:01.123952 5282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:01.125208 5267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:49:01.125361 5267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "d57ab8ec45fd4c49956f589a82d25bf0"
format_stamp: "Formatted at 2026-09-29 02:49:01 on dist-test-slave-t6c1"
server_key: "1b2e038335391e74d3dae4e396712db8"
server_key_iv: "4c103dcddbb31ae843b774aaa81ad932"
server_key_version: "encryptionkey@0"
I20260929 02:49:01.125550 5267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:01.142246 5267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:01.146590 5267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:01.146886 5267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:01.147933 5267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:01.149415 5267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:01.149499 5267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:01.149720 5267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:01.149799 5267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:01.181697 5267 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:43619
I20260929 02:49:01.181787 5394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:43619 every 8 connection(s)
I20260929 02:49:01.182955 5267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:01.184034 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 5267
I20260929 02:49:01.184247 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:01.184850 4886 external_mini_cluster.cc:1428] Setting key 310429a91f13345ef9f0cec9bc5b0792
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[4893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650141, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:01.203569 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:01.186226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:44305 (local address 127.4.197.190:42331)
0929 02:49:01.186419 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:01.186425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:01.187906 (+ 1481us) server_negotiation.cc:407] Connection header received
0929 02:49:01.189833 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:01.189837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:01.189903 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:01.190007 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:01.191432 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.192184 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.193703 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.194107 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.194192 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:01.197513 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:01.197518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:01.197522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:01.197556 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:01.200588 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.201412 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.201416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.201418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.201542 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.201951 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.201954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.201956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.202221 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:01.202326 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:01.203108 (+ 782us) server_negotiation.cc:299] Negotiation successful
0929 02:49:01.203300 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:49:01.204578 5397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:01.186859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42331 (local address 127.4.197.131:44305)
0929 02:49:01.187719 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:01.187779 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:01.189605 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:01.190193 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:01.190202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:01.190688 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:01.191293 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:01.191311 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.192322 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.192326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:01.193590 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:01.193602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.194295 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.194301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:01.194516 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:01.195174 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:01.195198 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:01.197324 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:01.200710 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:01.200721 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:01.200737 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:01.201298 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:01.201638 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:01.201641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:01.201644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:01.201847 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:01.202316 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:01.202323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:01.202697 (+ 374us) client_negotiation.cc:769] Sending connection context
0929 02:49:01.202983 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 02:49:01.203425 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":318,"threads_started":1}
I20260929 02:49:01.205943 5395 heartbeater.cc:347] Connected to a master server at 127.4.197.190:42331
I20260929 02:49:01.206254 5395 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:01.207130 5395 heartbeater.cc:511] Master 127.4.197.190:42331 requested a full tablet report, sending...
I20260929 02:49:01.208552 4938 ts_manager.cc:194] Registered new tserver with Master: d57ab8ec45fd4c49956f589a82d25bf0 (127.4.197.131:43619)
I20260929 02:49:01.209390 4938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:44305
I20260929 02:49:01.213220 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[4893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[4893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:01.255132 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:01.239498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.4.197.129:44435)
0929 02:49:01.240057 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:01.240064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:01.240124 (+ 60us) server_negotiation.cc:407] Connection header received
0929 02:49:01.240272 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:01.240279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:01.240504 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:01.240635 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:01.242038 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.243832 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.245114 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.245692 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.245794 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:01.249793 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:01.249809 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:01.249827 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:01.249877 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:01.252645 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.253300 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.253306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.253309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.253447 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.253751 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.253754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.253756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.254079 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:01.254291 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:01.254518 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 02:49:01.254715 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":289,"threads_started":1}
I20260929 02:49:01.255085 5128 heartbeater.cc:503] Master 127.4.197.190:42331 was elected leader, sending a full tablet report...
W20260929 02:49:01.256378 5043 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42182
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[4893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[4893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:01.297935 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:01.284354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.4.197.129:44435)
0929 02:49:01.284576 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:01.284583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:01.284828 (+ 245us) server_negotiation.cc:407] Connection header received
0929 02:49:01.284977 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:01.284982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:01.285086 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:01.285180 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:01.286390 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.287199 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.288481 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.288960 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.289064 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:01.293307 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:01.293312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:01.293315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:01.293364 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:01.295740 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.296374 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.296379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.296384 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.296536 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.296834 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.296839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.296842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.297092 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:01.297266 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:01.297490 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 02:49:01.297657 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:49:01.299398 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 5000
I20260929 02:49:01.311061 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:01.300405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.4.197.129:44435)
0929 02:49:01.300612 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:01.300617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:01.300779 (+ 162us) server_negotiation.cc:407] Connection header received
0929 02:49:01.300901 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:01.300906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:01.300964 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:01.301038 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:01.302109 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.302786 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.303983 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.304396 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.304492 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:01.306540 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:01.306545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:01.306548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:01.306577 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:01.309002 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.309559 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.309563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.309565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.309683 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.309951 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.309954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.309956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.310192 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:01.310366 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:01.310557 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 02:49:01.310711 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:49:01.311759 5123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:01.394441 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 5000
I20260929 02:49:01.406641 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 5134
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[4893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:49:01.421010 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:01.407534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.4.197.130:37327)
0929 02:49:01.408052 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:01.408061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:01.408086 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:49:01.408145 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:01.408151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:01.408355 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:01.408488 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:01.409494 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.410438 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.411698 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.412242 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.412345 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:01.415973 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:01.415984 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:01.415997 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:01.416028 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:01.418488 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.419157 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.419165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.419167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.419306 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.419657 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.419660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.419662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.420004 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:01.420244 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:01.420460 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 02:49:01.420663 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":245,"threads_started":1}
I20260929 02:49:01.422221 5257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:01.503700 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 5134
I20260929 02:49:01.515969 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 5267
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[4893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:49:01.531080 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:01.516744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.4.197.131:43619)
0929 02:49:01.517336 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:01.517343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:01.517371 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:49:01.517428 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:01.517436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:01.517642 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:01.517767 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:01.518905 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.519897 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.521094 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:01.521608 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:01.521701 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:01.525631 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:01.525644 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:01.525659 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:01.525693 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:01.528316 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.529011 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.529018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.529021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.529151 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:01.529508 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:01.529511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:01.529513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:01.529892 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:01.530124 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:01.530457 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 02:49:01.530664 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":287,"threads_started":1}
I20260929 02:49:01.532282 5390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:01.611845 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 5267
I20260929 02:49:01.623916 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 4906
I20260929 02:49:01.625121 4968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:01.726631 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 4906
2026-09-29T02:49:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3707 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[5416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[5416](info): set up 2 sockets
Sep 29 02:49:01 dist-test-slave-t6c1 krb5kdc[5416](info): commencing operation
krb5kdc: starting...
W20260929 02:49:02.723011 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:49:02 dist-test-slave-t6c1 krb5kdc[5416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:49:02.886664 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45315
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:45315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:03.200611 5429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:03.200937 5429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:03.201079 5429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:03.208578 5429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:03.208710 5429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:03.208801 5429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:03.208890 5429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:03.208950 5429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:03.219173 5429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:45315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:03.220525 5429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:03.221863 5429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:03.234087 5437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:03.234167 5435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:03.234066 5434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:03.235306 5429 server_base.cc:1034] running on GCE node
I20260929 02:49:03.236140 5429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:03.237651 5429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:03.238880 5429 hybrid_clock.cc:648] HybridClock initialized: now 1790650143238816 us; error 44 us; skew 500 ppm
I20260929 02:49:03.239257 5429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:03 dist-test-slave-t6c1 krb5kdc[5416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650143, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:03.245227 5429 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:49:03.246978 5429 webserver.cc:460] Webserver started at http://127.4.197.190:39517/ using document root <none> and password file <none>
I20260929 02:49:03.247699 5429 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:03.247828 5429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:03.248124 5429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:03.251502 5429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "4219f635c9214caf9e32105470fb420c"
format_stamp: "Formatted at 2026-09-29 02:49:03 on dist-test-slave-t6c1"
server_key: "14aaffc9aeb728e6c7961cd53e30dbc6"
server_key_iv: "fa60ec89c46e176963157bbc7c6d330a"
server_key_version: "encryptionkey@0"
I20260929 02:49:03.252280 5429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "4219f635c9214caf9e32105470fb420c"
format_stamp: "Formatted at 2026-09-29 02:49:03 on dist-test-slave-t6c1"
server_key: "14aaffc9aeb728e6c7961cd53e30dbc6"
server_key_iv: "fa60ec89c46e176963157bbc7c6d330a"
server_key_version: "encryptionkey@0"
I20260929 02:49:03.257823 5429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:49:03.261345 5444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:03.262746 5429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:49:03.262988 5429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "4219f635c9214caf9e32105470fb420c"
format_stamp: "Formatted at 2026-09-29 02:49:03 on dist-test-slave-t6c1"
server_key: "14aaffc9aeb728e6c7961cd53e30dbc6"
server_key_iv: "fa60ec89c46e176963157bbc7c6d330a"
server_key_version: "encryptionkey@0"
I20260929 02:49:03.263178 5429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:03.277384 5429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:03.281419 5429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:03.281677 5429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:03.298255 5429 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:45315
I20260929 02:49:03.298273 5495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:45315 every 8 connection(s)
I20260929 02:49:03.299585 5429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:03.302059 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 5429
I20260929 02:49:03.302284 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:03.303090 4886 external_mini_cluster.cc:1428] Setting key 3e80d5e3849d02ccedbc36ff141af1ec
I20260929 02:49:03.303633 5496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:03.314826 5496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c: Bootstrap starting.
Sep 29 02:49:03 dist-test-slave-t6c1 krb5kdc[5416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:03.317763 5496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:03.319018 5496 log.cc:824] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:03.321859 5496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c: No bootstrap required, opened a new log
I20260929 02:49:03.324689 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:03.305011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.4.197.190:45315)
0929 02:49:03.305672 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:03.305684 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:03.305719 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:49:03.307926 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:03.307945 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:03.308350 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:03.308686 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:03.310032 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:03.311305 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:03.312483 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:03.313166 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:03.313322 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:03.317121 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:03.317129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:03.317141 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:03.317174 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:03.320538 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:03.321123 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:03.321131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:03.321139 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:03.321332 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:03.321647 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:03.321652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:03.321655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:03.322299 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:03.322599 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:03.322972 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 02:49:03.323366 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":280,"threads_started":1}
I20260929 02:49:03.330474 5496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4219f635c9214caf9e32105470fb420c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45315 } }
I20260929 02:49:03.330830 5496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:03.330894 5496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4219f635c9214caf9e32105470fb420c, State: Initialized, Role: FOLLOWER
I20260929 02:49:03.331503 5496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4219f635c9214caf9e32105470fb420c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45315 } }
I20260929 02:49:03.331707 5496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:03.331831 5496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:03.331975 5496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:03.334049 5496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4219f635c9214caf9e32105470fb420c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45315 } }
I20260929 02:49:03.334542 5496 leader_election.cc:302] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4219f635c9214caf9e32105470fb420c; no voters:
I20260929 02:49:03.335150 5496 leader_election.cc:288] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:03.335274 5501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:03.335896 5501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [term 1 LEADER]: Becoming Leader. State: Replica: 4219f635c9214caf9e32105470fb420c, State: Running, Role: LEADER
I20260929 02:49:03.336405 5501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4219f635c9214caf9e32105470fb420c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45315 } }
I20260929 02:49:03.337083 5496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:03.339810 5502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4219f635c9214caf9e32105470fb420c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4219f635c9214caf9e32105470fb420c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45315 } } }
I20260929 02:49:03.339733 5503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4219f635c9214caf9e32105470fb420c. Latest consensus state: current_term: 1 leader_uuid: "4219f635c9214caf9e32105470fb420c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4219f635c9214caf9e32105470fb420c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45315 } } }
I20260929 02:49:03.340137 5502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:03.340744 5503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:03.341699 5509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:03.346704 5509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:03.355121 5509 catalog_manager.cc:1347] Generated new cluster ID: e7449a44398b48f2acb6122631ba1d69
I20260929 02:49:03.355254 5509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:03.366252 5509 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:03.368176 5509 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:03.382768 5509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4219f635c9214caf9e32105470fb420c: Generated new TSK 0
I20260929 02:49:03.383823 5509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:49:03.426704 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45315
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:03.740442 5522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:03.740711 5522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:03.740839 5522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:03.748328 5522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:03.748427 5522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:03.748595 5522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:03.758041 5522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45315
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:03.759409 5522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:03.760833 5522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:03.773725 5527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:03.773753 5528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:03.773952 5530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:03.775038 5522 server_base.cc:1034] running on GCE node
I20260929 02:49:03.775576 5522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:03.776469 5522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:03.777666 5522 hybrid_clock.cc:648] HybridClock initialized: now 1790650143777613 us; error 46 us; skew 500 ppm
I20260929 02:49:03.777942 5522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:03 dist-test-slave-t6c1 krb5kdc[5416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650143, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:03.780943 5522 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:49:03.782267 5522 webserver.cc:460] Webserver started at http://127.4.197.129:40653/ using document root <none> and password file <none>
I20260929 02:49:03.782953 5522 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:03.783066 5522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:03.783345 5522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:03.786108 5522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "11ddc887173a402a9abd1b80c0f5bef1"
format_stamp: "Formatted at 2026-09-29 02:49:03 on dist-test-slave-t6c1"
server_key: "db3384775d648408cdd05cf44575ee40"
server_key_iv: "d4bffb55bdf00661d4822930ab6e09d8"
server_key_version: "encryptionkey@0"
I20260929 02:49:03.786834 5522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "11ddc887173a402a9abd1b80c0f5bef1"
format_stamp: "Formatted at 2026-09-29 02:49:03 on dist-test-slave-t6c1"
server_key: "db3384775d648408cdd05cf44575ee40"
server_key_iv: "d4bffb55bdf00661d4822930ab6e09d8"
server_key_version: "encryptionkey@0"
I20260929 02:49:03.791600 5522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:49:03.794837 5537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:03.796155 5522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:49:03.796291 5522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "11ddc887173a402a9abd1b80c0f5bef1"
format_stamp: "Formatted at 2026-09-29 02:49:03 on dist-test-slave-t6c1"
server_key: "db3384775d648408cdd05cf44575ee40"
server_key_iv: "d4bffb55bdf00661d4822930ab6e09d8"
server_key_version: "encryptionkey@0"
I20260929 02:49:03.796466 5522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:03.818082 5522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:03.822389 5522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:03.822661 5522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:03.823698 5522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:03.824990 5522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:03.825071 5522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:03.825186 5522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:03.825237 5522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:03.854650 5522 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:37035
I20260929 02:49:03.854692 5649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:37035 every 8 connection(s)
I20260929 02:49:03.855983 5522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:03.862246 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 5522
I20260929 02:49:03.862438 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:03.863080 4886 external_mini_cluster.cc:1428] Setting key f119ae5d774eae22e7fa76de6f5fc46a
Sep 29 02:49:03 dist-test-slave-t6c1 krb5kdc[5416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650143, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:03.876631 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:03.858380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:41423 (local address 127.4.197.190:45315)
0929 02:49:03.858806 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:03.858810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:03.859770 (+ 960us) server_negotiation.cc:407] Connection header received
0929 02:49:03.861615 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:03.861621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:03.861710 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:03.861805 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:03.863589 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:03.864322 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:03.865893 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:03.866289 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:03.866399 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:03.870107 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:03.870113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:03.870117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:03.870159 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:03.873267 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:03.874097 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:03.874104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:03.874108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:03.874259 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:03.874652 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:03.874657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:03.874661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:03.875075 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:03.875229 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:03.875983 (+ 754us) server_negotiation.cc:299] Negotiation successful
0929 02:49:03.876238 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":220,"threads_started":1}
I20260929 02:49:03.877471 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:03.858844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45315 (local address 127.4.197.129:41423)
0929 02:49:03.859589 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:03.859628 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:03.861327 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:03.862089 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:03.862098 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:03.862717 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:03.863402 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:03.863424 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:03.864475 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:03.864479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:03.865752 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:03.865765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:03.866511 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:03.866519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:03.866776 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:03.867497 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:03.867523 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:03.869854 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:03.873449 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:03.873457 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:03.873472 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:03.873965 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:03.874353 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:03.874356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:03.874359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:03.874545 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:03.875215 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:03.875221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:03.875580 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 02:49:03.875816 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 02:49:03.876246 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":272,"threads_started":1}
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:49:03.879094 5650 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45315
I20260929 02:49:03.879392 5650 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:03.880148 5650 heartbeater.cc:511] Master 127.4.197.190:45315 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
I20260929 02:49:03.882543 5461 ts_manager.cc:194] Registered new tserver with Master: 11ddc887173a402a9abd1b80c0f5bef1 (127.4.197.129:37035)
I20260929 02:49:03.884524 5461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:41423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:49:03.901976 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45315
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:04.207131 5656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:04.207437 5656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:04.207579 5656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:04.214668 5656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:04.214781 5656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:04.214951 5656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:04.224501 5656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45315
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:04.225850 5656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:04.227209 5656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:04.239779 5661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:04.239822 5662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:04.239759 5664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:04.240819 5656 server_base.cc:1034] running on GCE node
I20260929 02:49:04.241525 5656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:04.242595 5656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:04.243799 5656 hybrid_clock.cc:648] HybridClock initialized: now 1790650144243722 us; error 70 us; skew 500 ppm
I20260929 02:49:04.244132 5656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:04 dist-test-slave-t6c1 krb5kdc[5416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:04.247246 5656 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:49:04.248735 5656 webserver.cc:460] Webserver started at http://127.4.197.130:40403/ using document root <none> and password file <none>
I20260929 02:49:04.249435 5656 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:04.249550 5656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:04.249841 5656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:04.252813 5656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "7ee90527865548c3a1a63fccb0f64413"
format_stamp: "Formatted at 2026-09-29 02:49:04 on dist-test-slave-t6c1"
server_key: "a0abe060af9293fa868a3d04adac7c5a"
server_key_iv: "3fcd83fb1266adba7096724ed5148826"
server_key_version: "encryptionkey@0"
I20260929 02:49:04.253623 5656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "7ee90527865548c3a1a63fccb0f64413"
format_stamp: "Formatted at 2026-09-29 02:49:04 on dist-test-slave-t6c1"
server_key: "a0abe060af9293fa868a3d04adac7c5a"
server_key_iv: "3fcd83fb1266adba7096724ed5148826"
server_key_version: "encryptionkey@0"
I20260929 02:49:04.258862 5656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:49:04.262324 5671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:04.263697 5656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:04.263846 5656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "7ee90527865548c3a1a63fccb0f64413"
format_stamp: "Formatted at 2026-09-29 02:49:04 on dist-test-slave-t6c1"
server_key: "a0abe060af9293fa868a3d04adac7c5a"
server_key_iv: "3fcd83fb1266adba7096724ed5148826"
server_key_version: "encryptionkey@0"
I20260929 02:49:04.264022 5656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:04.279184 5656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:04.283481 5656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:04.283825 5656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:04.284905 5656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:04.286540 5656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:04.286625 5656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:04.286767 5656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:04.286832 5656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:04.325943 5656 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:39973
I20260929 02:49:04.326000 5783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:39973 every 8 connection(s)
I20260929 02:49:04.327234 5656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:04.336509 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 5656
I20260929 02:49:04.336691 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:04.337224 4886 external_mini_cluster.cc:1428] Setting key 8a81ca4a85b8b9d0aca0172e87865670
Sep 29 02:49:04 dist-test-slave-t6c1 krb5kdc[5416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:04.348115 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.329794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:36289 (local address 127.4.197.190:45315)
0929 02:49:04.329978 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:04.329982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:04.331116 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 02:49:04.332890 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:04.332894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:04.332969 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:04.333045 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:04.334562 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.335396 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.337096 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.337736 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.337914 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.341605 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:04.341613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:04.341617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:04.341661 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:04.344792 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:04.345636 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:04.345640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:04.345643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:04.345802 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:04.346285 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:04.346288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:04.346290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:04.346578 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:04.346707 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:04.347614 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 02:49:04.347849 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:49:04.349530 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.330242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45315 (local address 127.4.197.130:36289)
0929 02:49:04.330967 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:04.331010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:04.332623 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:04.333268 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:04.333276 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:04.333784 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:04.334373 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.334391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.335555 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.335560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.336961 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.336973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.337994 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.338002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.338290 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.338958 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:04.338987 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:04.341358 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:04.344949 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:04.344957 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:04.344977 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:04.345525 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:04.345931 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:04.345936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:04.345940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:04.346175 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:04.346695 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:04.346703 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:04.347200 (+ 497us) client_negotiation.cc:769] Sending connection context
0929 02:49:04.347505 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 02:49:04.348061 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":264,"threads_started":1}
I20260929 02:49:04.351310 5784 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45315
I20260929 02:49:04.351648 5784 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:04.352468 5784 heartbeater.cc:511] Master 127.4.197.190:45315 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
I20260929 02:49:04.354102 5461 ts_manager.cc:194] Registered new tserver with Master: 7ee90527865548c3a1a63fccb0f64413 (127.4.197.130:39973)
I20260929 02:49:04.354982 5461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:36289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:49:04.375944 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45315
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:04.686506 5789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:04.686794 5789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:04.686893 5789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:04.694240 5789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:04.694356 5789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:04.694475 5789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:04.704331 5789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45315
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:04.705514 5789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:04.706831 5789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:04.718782 5794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:04.718736 5795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:04.718804 5797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:04.719564 5789 server_base.cc:1034] running on GCE node
I20260929 02:49:04.720105 5789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:04.720928 5789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:04.722126 5789 hybrid_clock.cc:648] HybridClock initialized: now 1790650144722073 us; error 43 us; skew 500 ppm
I20260929 02:49:04.722416 5789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:04 dist-test-slave-t6c1 krb5kdc[5416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:04.725441 5789 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:49:04.726956 5789 webserver.cc:460] Webserver started at http://127.4.197.131:40839/ using document root <none> and password file <none>
I20260929 02:49:04.727640 5789 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:04.727746 5789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:04.728056 5789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:04.731143 5789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "57c291d151a84b47aa7c29bd0c108fb7"
format_stamp: "Formatted at 2026-09-29 02:49:04 on dist-test-slave-t6c1"
server_key: "9086cab5524dc6a2726a841656e45c1a"
server_key_iv: "e70b5707d3a94e176fce47159b26fe4a"
server_key_version: "encryptionkey@0"
I20260929 02:49:04.731920 5789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "57c291d151a84b47aa7c29bd0c108fb7"
format_stamp: "Formatted at 2026-09-29 02:49:04 on dist-test-slave-t6c1"
server_key: "9086cab5524dc6a2726a841656e45c1a"
server_key_iv: "e70b5707d3a94e176fce47159b26fe4a"
server_key_version: "encryptionkey@0"
I20260929 02:49:04.737272 5789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:49:04.740959 5804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:04.742448 5789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:49:04.742599 5789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "57c291d151a84b47aa7c29bd0c108fb7"
format_stamp: "Formatted at 2026-09-29 02:49:04 on dist-test-slave-t6c1"
server_key: "9086cab5524dc6a2726a841656e45c1a"
server_key_iv: "e70b5707d3a94e176fce47159b26fe4a"
server_key_version: "encryptionkey@0"
I20260929 02:49:04.742767 5789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:04.766184 5789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:04.770872 5789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:04.771188 5789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:04.772156 5789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:04.773449 5789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:04.773524 5789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:04.773622 5789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:04.773663 5789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:04.803011 5789 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:34693
I20260929 02:49:04.803093 5916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:34693 every 8 connection(s)
I20260929 02:49:04.804278 5789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:04.811935 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 5789
I20260929 02:49:04.812105 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:04.812495 4886 external_mini_cluster.cc:1428] Setting key baace09f7867ec885840ae3c7cce7630
Sep 29 02:49:04 dist-test-slave-t6c1 krb5kdc[5416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650144, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:04.823006 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.806666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:49027 (local address 127.4.197.190:45315)
0929 02:49:04.806840 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:04.806844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:04.808075 (+ 1231us) server_negotiation.cc:407] Connection header received
0929 02:49:04.809853 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:04.809858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:04.809942 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:04.810027 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:04.811519 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.812342 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.814253 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.814697 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.814809 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.818038 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:04.818042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:04.818045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:04.818076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:04.820650 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:04.821245 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:04.821248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:04.821251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:04.821355 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:04.821660 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:04.821662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:04.821664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:04.821927 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:04.822028 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:04.822572 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 02:49:04.822727 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:49:04.823894 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.807212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45315 (local address 127.4.197.131:49027)
0929 02:49:04.807914 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:04.807959 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:04.809591 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:04.810264 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:04.810272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:04.810770 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:04.811360 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.811378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.812487 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.812490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.814124 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.814137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.814899 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.814906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.815202 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.815817 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:04.815837 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:04.817848 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:04.820771 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:04.820777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:04.820787 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:04.821160 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:04.821430 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:04.821433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:04.821435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:04.821586 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:04.822008 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:04.822013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:04.822280 (+ 267us) client_negotiation.cc:769] Sending connection context
0929 02:49:04.822494 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 02:49:04.822839 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":256,"threads_started":1}
I20260929 02:49:04.825085 5917 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45315
I20260929 02:49:04.825351 5917 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:04.826059 5917 heartbeater.cc:511] Master 127.4.197.190:45315 requested a full tablet report, sending...
I20260929 02:49:04.827435 5461 ts_manager.cc:194] Registered new tserver with Master: 57c291d151a84b47aa7c29bd0c108fb7 (127.4.197.131:34693)
I20260929 02:49:04.828163 5461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:49027
I20260929 02:49:04.839172 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:04.857112 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.845106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.4.197.190:45315)
0929 02:49:04.845299 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:04.845305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:04.845686 (+ 381us) server_negotiation.cc:407] Connection header received
0929 02:49:04.845812 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:04.845816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:04.845875 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:04.845965 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:04.847410 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.848138 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.849704 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.850083 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.850188 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.852357 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:04.852362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:04.852366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:04.852397 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:04.854645 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:04.855299 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:04.855303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:04.855305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:04.855427 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:04.855762 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:04.855765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:04.855767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:04.856066 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:04.856276 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:04.856624 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 02:49:04.856832 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:49:04.862766 5461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:04.869470 5461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:04.888244 5650 heartbeater.cc:503] Master 127.4.197.190:45315 was elected leader, sending a full tablet report...
I20260929 02:49:04.905014 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.881456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:37035 (local address 127.0.0.1:47836)
0929 02:49:04.894659 (+ 13203us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:04.894720 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:04.895413 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:04.896452 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:04.896460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:04.896529 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:04.897895 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.897904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.899972 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.899976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.902611 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.902621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.903942 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.903948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.904124 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.904172 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 02:49:04.904325 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 02:49:04.904514 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12867,"mutex_wait_us":196,"spinlock_wait_cycles":34560,"thread_start_us":13220,"threads_started":1}
I20260929 02:49:04.905113 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.881822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:39973 (local address 127.0.0.1:54810)
0929 02:49:04.894966 (+ 13144us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:04.894985 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:04.895175 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:04.896464 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:04.896468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:04.896505 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:04.897529 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.897542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.899822 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.899828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.902569 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.902588 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.904335 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.904341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.904503 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.904530 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 02:49:04.904618 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:49:04.904741 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12996,"mutex_wait_us":219,"spinlock_wait_cycles":1280,"thread_start_us":13129,"threads_started":1}
I20260929 02:49:04.905694 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.882744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.4.197.130:39973)
0929 02:49:04.895878 (+ 13134us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:04.895887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:04.895929 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:49:04.896025 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:04.896033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:04.896273 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:04.898336 (+ 2063us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:04.898533 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.899699 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.902754 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.904138 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.904278 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.904460 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:04.904675 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 02:49:04.904839 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12974,"thread_start_us":12880,"threads_started":1}
I20260929 02:49:04.906109 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.881622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.4.197.129:37035)
0929 02:49:04.894049 (+ 12427us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:04.894060 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:04.894898 (+ 838us) server_negotiation.cc:407] Connection header received
0929 02:49:04.895611 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:04.895622 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:04.895928 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:04.896132 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:04.898069 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.899781 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.902770 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.903755 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.903902 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.904140 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:04.905275 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0929 02:49:04.905461 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12183,"thread_start_us":12449,"threads_started":1}
I20260929 02:49:04.906140 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.882038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:34693 (local address 127.0.0.1:34674)
0929 02:49:04.896403 (+ 14365us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:04.896420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:04.896553 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:04.897252 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:04.897256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:04.897282 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:04.897830 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.897841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.899923 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.899931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.902315 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.902328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.905387 (+ 3059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.905393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.905548 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.905579 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:49:04.905646 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 02:49:04.905770 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14240,"mutex_wait_us":253,"spinlock_wait_cycles":896,"thread_start_us":14115,"threads_started":1}
I20260929 02:49:04.907070 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.883164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.4.197.131:34693)
0929 02:49:04.895955 (+ 12791us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:04.895963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:04.896695 (+ 732us) server_negotiation.cc:407] Connection header received
0929 02:49:04.896800 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:04.896822 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:04.897066 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:04.897266 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:04.898061 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.899788 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.902475 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.903861 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.904004 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.906475 (+ 2471us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:04.906542 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 02:49:04.906702 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12598,"thread_start_us":12662,"threads_started":1}
I20260929 02:49:04.909925 5719 tablet_service.cc:1467] Processing CreateTablet for tablet 56402a6ca29049ba8cc71d2eb570d4c5 (DEFAULT_TABLE table=test-table [id=6e2b7b513b514186b99d9ceb4170ef5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:04.909744 5585 tablet_service.cc:1467] Processing CreateTablet for tablet 56402a6ca29049ba8cc71d2eb570d4c5 (DEFAULT_TABLE table=test-table [id=6e2b7b513b514186b99d9ceb4170ef5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:04.910261 5852 tablet_service.cc:1467] Processing CreateTablet for tablet 56402a6ca29049ba8cc71d2eb570d4c5 (DEFAULT_TABLE table=test-table [id=6e2b7b513b514186b99d9ceb4170ef5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:04.911105 5719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56402a6ca29049ba8cc71d2eb570d4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:04.911180 5585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56402a6ca29049ba8cc71d2eb570d4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:04.911516 5852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56402a6ca29049ba8cc71d2eb570d4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:04.922389 5935 tablet_bootstrap.cc:492] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413: Bootstrap starting.
I20260929 02:49:04.924870 5936 tablet_bootstrap.cc:492] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7: Bootstrap starting.
I20260929 02:49:04.927786 5935 tablet_bootstrap.cc:654] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:04.928426 5936 tablet_bootstrap.cc:654] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:04.929106 5935 log.cc:824] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:04.929610 5936 log.cc:824] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:04.930720 5937 tablet_bootstrap.cc:492] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1: Bootstrap starting.
I20260929 02:49:04.935191 5937 tablet_bootstrap.cc:654] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:04.935513 5935 tablet_bootstrap.cc:492] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413: No bootstrap required, opened a new log
I20260929 02:49:04.935782 5935 ts_tablet_manager.cc:1397] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260929 02:49:04.936451 5937 log.cc:824] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:04.937275 5936 tablet_bootstrap.cc:492] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7: No bootstrap required, opened a new log
I20260929 02:49:04.937690 5936 ts_tablet_manager.cc:1397] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260929 02:49:04.939812 5937 tablet_bootstrap.cc:492] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1: No bootstrap required, opened a new log
I20260929 02:49:04.940114 5937 ts_tablet_manager.cc:1397] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260929 02:49:04.944456 5935 raft_consensus.cc:348] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ddc887173a402a9abd1b80c0f5bef1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 37035 } } peers { permanent_uuid: "7ee90527865548c3a1a63fccb0f64413" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39973 } } peers { permanent_uuid: "57c291d151a84b47aa7c29bd0c108fb7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 34693 } }
I20260929 02:49:04.944871 5935 raft_consensus.cc:374] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:04.944968 5935 raft_consensus.cc:729] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee90527865548c3a1a63fccb0f64413, State: Initialized, Role: FOLLOWER
I20260929 02:49:04.945608 5935 consensus_queue.cc:260] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ddc887173a402a9abd1b80c0f5bef1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 37035 } } peers { permanent_uuid: "7ee90527865548c3a1a63fccb0f64413" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39973 } } peers { permanent_uuid: "57c291d151a84b47aa7c29bd0c108fb7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 34693 } }
I20260929 02:49:04.946704 5784 heartbeater.cc:503] Master 127.4.197.190:45315 was elected leader, sending a full tablet report...
I20260929 02:49:04.947490 5935 ts_tablet_manager.cc:1428] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260929 02:49:04.949752 5936 raft_consensus.cc:348] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ddc887173a402a9abd1b80c0f5bef1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 37035 } } peers { permanent_uuid: "7ee90527865548c3a1a63fccb0f64413" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39973 } } peers { permanent_uuid: "57c291d151a84b47aa7c29bd0c108fb7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 34693 } }
I20260929 02:49:04.950122 5936 raft_consensus.cc:374] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:04.950223 5936 raft_consensus.cc:729] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c291d151a84b47aa7c29bd0c108fb7, State: Initialized, Role: FOLLOWER
I20260929 02:49:04.950811 5936 consensus_queue.cc:260] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ddc887173a402a9abd1b80c0f5bef1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 37035 } } peers { permanent_uuid: "7ee90527865548c3a1a63fccb0f64413" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39973 } } peers { permanent_uuid: "57c291d151a84b47aa7c29bd0c108fb7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 34693 } }
I20260929 02:49:04.951269 5937 raft_consensus.cc:348] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ddc887173a402a9abd1b80c0f5bef1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 37035 } } peers { permanent_uuid: "7ee90527865548c3a1a63fccb0f64413" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39973 } } peers { permanent_uuid: "57c291d151a84b47aa7c29bd0c108fb7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 34693 } }
I20260929 02:49:04.951705 5937 raft_consensus.cc:374] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:04.951808 5937 raft_consensus.cc:729] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ddc887173a402a9abd1b80c0f5bef1, State: Initialized, Role: FOLLOWER
I20260929 02:49:04.952486 5936 ts_tablet_manager.cc:1428] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260929 02:49:04.952401 5937 consensus_queue.cc:260] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ddc887173a402a9abd1b80c0f5bef1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 37035 } } peers { permanent_uuid: "7ee90527865548c3a1a63fccb0f64413" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39973 } } peers { permanent_uuid: "57c291d151a84b47aa7c29bd0c108fb7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 34693 } }
I20260929 02:49:04.952628 5917 heartbeater.cc:503] Master 127.4.197.190:45315 was elected leader, sending a full tablet report...
I20260929 02:49:04.955379 5937 ts_tablet_manager.cc:1428] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260929 02:49:04.960011 5942 raft_consensus.cc:482] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:04.960212 5942 raft_consensus.cc:504] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ddc887173a402a9abd1b80c0f5bef1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 37035 } } peers { permanent_uuid: "7ee90527865548c3a1a63fccb0f64413" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39973 } } peers { permanent_uuid: "57c291d151a84b47aa7c29bd0c108fb7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 34693 } }
I20260929 02:49:04.961658 5942 leader_election.cc:288] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11ddc887173a402a9abd1b80c0f5bef1 (127.4.197.129:37035), 7ee90527865548c3a1a63fccb0f64413 (127.4.197.130:39973)
I20260929 02:49:04.969070 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.962479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:58599 (local address 127.4.197.130:39973)
0929 02:49:04.962703 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:04.962709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:04.962728 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:49:04.962783 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:04.962786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:04.962837 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:04.962951 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:04.963303 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.964218 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.966984 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.968206 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.968354 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.968423 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:04.968700 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 02:49:04.968827 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:49:04.969128 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.962026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:39973 (local address 127.4.197.131:58599)
0929 02:49:04.962231 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:04.962253 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:04.962375 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:04.962993 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:04.962997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:04.963022 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:04.963170 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.963179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.964365 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.964369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.966792 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.966807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.968345 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.968349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.968494 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.968589 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 02:49:04.968689 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:49:04.968839 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260929 02:49:04.969269 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.962136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:59849 (local address 127.4.197.129:37035)
0929 02:49:04.962321 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:04.962342 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:04.962898 (+ 556us) server_negotiation.cc:407] Connection header received
0929 02:49:04.963070 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:04.963076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:04.963132 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:04.963197 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:04.963681 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.964562 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.967152 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.968385 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.968524 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.968585 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:04.968952 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 02:49:04.969069 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:49:04.969605 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.962167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:37035 (local address 127.4.197.131:59849)
0929 02:49:04.962802 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:04.962816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:04.962946 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:04.963237 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:04.963242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:04.963264 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:04.963542 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.963550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.965009 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.965014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.967032 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.967042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.968620 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.968625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.968786 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.968825 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:49:04.968898 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 02:49:04.969016 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":1,"thread_start_us":380,"threads_started":1}
I20260929 02:49:04.970095 5739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56402a6ca29049ba8cc71d2eb570d4c5" candidate_uuid: "57c291d151a84b47aa7c29bd0c108fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ee90527865548c3a1a63fccb0f64413" is_pre_election: true
I20260929 02:49:04.970464 5605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56402a6ca29049ba8cc71d2eb570d4c5" candidate_uuid: "57c291d151a84b47aa7c29bd0c108fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ddc887173a402a9abd1b80c0f5bef1" is_pre_election: true
I20260929 02:49:04.970619 5739 raft_consensus.cc:2454] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c291d151a84b47aa7c29bd0c108fb7 in term 0.
I20260929 02:49:04.970911 5605 raft_consensus.cc:2454] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c291d151a84b47aa7c29bd0c108fb7 in term 0.
I20260929 02:49:04.971268 5806 leader_election.cc:302] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57c291d151a84b47aa7c29bd0c108fb7, 7ee90527865548c3a1a63fccb0f64413; no voters:
I20260929 02:49:04.971688 5942 raft_consensus.cc:2781] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:04.971814 5942 raft_consensus.cc:482] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:04.971922 5942 raft_consensus.cc:3037] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:04.974162 5942 raft_consensus.cc:504] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ddc887173a402a9abd1b80c0f5bef1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 37035 } } peers { permanent_uuid: "7ee90527865548c3a1a63fccb0f64413" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39973 } } peers { permanent_uuid: "57c291d151a84b47aa7c29bd0c108fb7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 34693 } }
I20260929 02:49:04.974803 5942 leader_election.cc:288] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [CANDIDATE]: Term 1 election: Requested vote from peers 11ddc887173a402a9abd1b80c0f5bef1 (127.4.197.129:37035), 7ee90527865548c3a1a63fccb0f64413 (127.4.197.130:39973)
I20260929 02:49:04.975415 5605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56402a6ca29049ba8cc71d2eb570d4c5" candidate_uuid: "57c291d151a84b47aa7c29bd0c108fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ddc887173a402a9abd1b80c0f5bef1"
I20260929 02:49:04.975415 5739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56402a6ca29049ba8cc71d2eb570d4c5" candidate_uuid: "57c291d151a84b47aa7c29bd0c108fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ee90527865548c3a1a63fccb0f64413"
I20260929 02:49:04.975621 5605 raft_consensus.cc:3037] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:04.975612 5739 raft_consensus.cc:3037] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:04.977878 5605 raft_consensus.cc:2454] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c291d151a84b47aa7c29bd0c108fb7 in term 1.
I20260929 02:49:04.978375 5739 raft_consensus.cc:2454] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c291d151a84b47aa7c29bd0c108fb7 in term 1.
I20260929 02:49:04.978437 5808 leader_election.cc:302] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ddc887173a402a9abd1b80c0f5bef1, 57c291d151a84b47aa7c29bd0c108fb7; no voters:
I20260929 02:49:04.978804 5942 raft_consensus.cc:2781] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:04.979274 5942 raft_consensus.cc:686] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [term 1 LEADER]: Becoming Leader. State: Replica: 57c291d151a84b47aa7c29bd0c108fb7, State: Running, Role: LEADER
I20260929 02:49:04.979810 5942 consensus_queue.cc:237] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ddc887173a402a9abd1b80c0f5bef1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 37035 } } peers { permanent_uuid: "7ee90527865548c3a1a63fccb0f64413" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39973 } } peers { permanent_uuid: "57c291d151a84b47aa7c29bd0c108fb7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 34693 } }
I20260929 02:49:04.984282 5461 catalog_manager.cc:5495] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57c291d151a84b47aa7c29bd0c108fb7 (127.4.197.131). New cstate: current_term: 1 leader_uuid: "57c291d151a84b47aa7c29bd0c108fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ddc887173a402a9abd1b80c0f5bef1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 37035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ee90527865548c3a1a63fccb0f64413" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57c291d151a84b47aa7c29bd0c108fb7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 34693 } health_report { overall_health: HEALTHY } } }
Sep 29 02:49:05 dist-test-slave-t6c1 krb5kdc[5416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:05.018855 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:05.004976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.4.197.129:37035)
0929 02:49:05.005137 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:05.005142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:05.005634 (+ 492us) server_negotiation.cc:407] Connection header received
0929 02:49:05.005760 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:05.005763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:05.005827 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:05.005920 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:05.007436 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.008336 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.009590 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.010026 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.010136 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:05.014116 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:05.014130 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:05.014145 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:05.014179 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:05.016838 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.017391 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.017398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.017400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.017512 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.017765 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.017767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.017769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.018079 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:05.018286 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:05.018451 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 02:49:05.018607 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 02:49:05 dist-test-slave-t6c1 krb5kdc[5416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650145, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 02:49:05.055778 5918 tablet.cc:2369] T 56402a6ca29049ba8cc71d2eb570d4c5 P 57c291d151a84b47aa7c29bd0c108fb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:49:05 dist-test-slave-t6c1 krb5kdc[5416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650145, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:05.062685 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:05.048818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.4.197.129:37035)
0929 02:49:05.049021 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:05.049027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:05.049265 (+ 238us) server_negotiation.cc:407] Connection header received
0929 02:49:05.049440 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:05.049444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:05.049508 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:05.049593 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:05.050831 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.051610 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.053261 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.053667 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.053786 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:05.057585 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:05.057590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:05.057593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:05.057626 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:05.060364 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.060908 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.060913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.060917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.061038 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.061410 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.061414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.061416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.061653 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:05.061899 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:05.062155 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 02:49:05.062374 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 02:49:05.079326 5785 tablet.cc:2369] T 56402a6ca29049ba8cc71d2eb570d4c5 P 7ee90527865548c3a1a63fccb0f64413: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:49:05 dist-test-slave-t6c1 krb5kdc[5416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:49:05 dist-test-slave-t6c1 krb5kdc[5416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:05.105643 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:05.091558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.4.197.129:37035)
0929 02:49:05.091766 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:05.091774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:05.092065 (+ 291us) server_negotiation.cc:407] Connection header received
0929 02:49:05.092237 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:05.092242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:05.092306 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:05.092393 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:05.093742 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.094438 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.095680 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.096116 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.096228 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:05.100185 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:05.100190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:05.100193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:05.100230 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:05.102695 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.103374 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.103380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.103383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.103507 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.103866 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.103870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.103872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.104162 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:05.104390 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:05.104678 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 02:49:05.104908 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 02:49:05.107537 5565 tablet_service.cc:2908] Rejecting scan request for tablet 56402a6ca29049ba8cc71d2eb570d4c5: Uninitialized: clean time has not yet been initialized
W20260929 02:49:05.108079 5651 tablet.cc:2369] T 56402a6ca29049ba8cc71d2eb570d4c5 P 11ddc887173a402a9abd1b80c0f5bef1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:05.110692 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 5522
I20260929 02:49:05.122457 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:05.111493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.4.197.129:37035)
0929 02:49:05.111640 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:05.111644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:05.112015 (+ 371us) server_negotiation.cc:407] Connection header received
0929 02:49:05.112083 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:05.112089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:05.112177 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:05.112271 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:05.113624 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.114523 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.115764 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.116159 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.116257 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:05.118298 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:05.118302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:05.118305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:05.118338 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:05.120541 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.121079 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.121082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.121085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.121196 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.121444 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.121446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.121448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.121688 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:05.121912 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:05.122074 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 02:49:05.122235 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 02:49:05.123111 5645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:05.209748 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 5522
I20260929 02:49:05.222133 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 5656
Sep 29 02:49:05 dist-test-slave-t6c1 krb5kdc[5416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:49:05.236318 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:05.222989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.4.197.130:39973)
0929 02:49:05.223140 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:05.223144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:05.223161 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:49:05.223244 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:05.223248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:05.223325 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:05.223404 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:05.224537 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.225334 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.226548 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.226992 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.227103 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:05.230963 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:05.230983 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:05.230998 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:05.231040 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:05.233593 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.234286 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.234293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.234296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.234446 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.234814 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.234818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.234820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.235367 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:05.235609 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:05.235906 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 02:49:05.236074 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:49:05.236989 5779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:05.321298 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 5656
I20260929 02:49:05.333727 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 5789
Sep 29 02:49:05 dist-test-slave-t6c1 krb5kdc[5416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:49:05.347290 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:05.334440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.4.197.131:34693)
0929 02:49:05.334631 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:05.334635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:05.334653 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:49:05.334781 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:05.334785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:05.334857 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:05.334959 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:05.336064 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.336769 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.337993 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.338424 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.338539 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:05.342325 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:05.342340 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:05.342355 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:05.342389 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:05.344966 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.345580 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.345587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.345590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.345714 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.345976 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.345979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.345981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.346316 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:05.346530 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:05.346779 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 02:49:05.346990 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:49:05.347924 5912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:05.435829 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 5789
I20260929 02:49:05.448472 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 5429
I20260929 02:49:05.449543 5491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:05.556180 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 5429
2026-09-29T02:49:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3826 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:05 dist-test-slave-t6c1 krb5kdc[5964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:05 dist-test-slave-t6c1 krb5kdc[5964](info): set up 2 sockets
Sep 29 02:49:05 dist-test-slave-t6c1 krb5kdc[5964](info): commencing operation
krb5kdc: starting...
W20260929 02:49:06.546980 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:49:06 dist-test-slave-t6c1 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:49:06.710770 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:39073
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:39073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:07.015300 5977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:07.015597 5977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:07.015692 5977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:07.023065 5977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:07.023181 5977 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:49:07.023221 5977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:07.023280 5977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:07.023337 5977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:07.023419 5977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:07.033318 5977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:39073
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:39073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:07.034592 5977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:07.035876 5977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:07.047189 5982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:07.047215 5983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:07.047364 5985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:07.047947 5977 server_base.cc:1034] running on GCE node
I20260929 02:49:07.048871 5977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:07.050371 5977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:07.051594 5977 hybrid_clock.cc:648] HybridClock initialized: now 1790650147051538 us; error 32 us; skew 500 ppm
I20260929 02:49:07.051914 5977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:07 dist-test-slave-t6c1 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650147, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:07.055099 5977 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:49:07.056527 5977 webserver.cc:460] Webserver started at http://127.4.197.190:33395/ using document root <none> and password file <none>
I20260929 02:49:07.057226 5977 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:07.057325 5977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:07.057636 5977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:07.060655 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "9697cda8f1a846d1a60c89457cbf9a27"
format_stamp: "Formatted at 2026-09-29 02:49:07 on dist-test-slave-t6c1"
server_key: "8e2c6e9138d861ace98cf307ff298143"
server_key_iv: "ccd3c542ce6e2e907851d8efb3609aa4"
server_key_version: "encryptionkey@0"
I20260929 02:49:07.061440 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "9697cda8f1a846d1a60c89457cbf9a27"
format_stamp: "Formatted at 2026-09-29 02:49:07 on dist-test-slave-t6c1"
server_key: "8e2c6e9138d861ace98cf307ff298143"
server_key_iv: "ccd3c542ce6e2e907851d8efb3609aa4"
server_key_version: "encryptionkey@0"
I20260929 02:49:07.066833 5977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:49:07.070385 5992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:07.071863 5977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:49:07.072022 5977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "9697cda8f1a846d1a60c89457cbf9a27"
format_stamp: "Formatted at 2026-09-29 02:49:07 on dist-test-slave-t6c1"
server_key: "8e2c6e9138d861ace98cf307ff298143"
server_key_iv: "ccd3c542ce6e2e907851d8efb3609aa4"
server_key_version: "encryptionkey@0"
I20260929 02:49:07.072220 5977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:07.101794 5977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:07.105721 5977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:07.105943 5977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:07.123720 5977 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:39073
I20260929 02:49:07.123741 6053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:39073 every 8 connection(s)
I20260929 02:49:07.124971 5977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:07.125452 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 5977
I20260929 02:49:07.125684 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:07.126557 4886 external_mini_cluster.cc:1428] Setting key a40644bb12f24b86c3a6d92dd503ab69
I20260929 02:49:07.128949 6054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:07.142756 6054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27: Bootstrap starting.
Sep 29 02:49:07 dist-test-slave-t6c1 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:07.146705 6054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:07.148078 6054 log.cc:824] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:07.150888 6054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27: No bootstrap required, opened a new log
I20260929 02:49:07.152237 6057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:07.128752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.4.197.190:39073)
0929 02:49:07.131443 (+ 2691us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:07.131454 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:07.131501 (+ 47us) server_negotiation.cc:407] Connection header received
0929 02:49:07.133215 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:07.133236 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:07.133644 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:07.133983 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:07.135746 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:07.137011 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:07.138687 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:07.139475 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:07.139626 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:07.144407 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:07.144419 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:07.144441 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:07.144500 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:07.148200 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:07.148902 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:07.148912 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:07.148919 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:07.149110 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:07.149481 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:07.149485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:07.149487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:07.150034 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:07.150302 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:07.150768 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 02:49:07.151248 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2470,"thread_start_us":2216,"threads_started":1}
I20260929 02:49:07.158183 6054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9697cda8f1a846d1a60c89457cbf9a27" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 39073 } }
I20260929 02:49:07.158532 6054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:07.158619 6054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9697cda8f1a846d1a60c89457cbf9a27, State: Initialized, Role: FOLLOWER
I20260929 02:49:07.159190 6054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9697cda8f1a846d1a60c89457cbf9a27" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 39073 } }
I20260929 02:49:07.159399 6054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:07.159512 6054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:07.159654 6054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:07.161763 6054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9697cda8f1a846d1a60c89457cbf9a27" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 39073 } }
I20260929 02:49:07.162238 6054 leader_election.cc:302] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9697cda8f1a846d1a60c89457cbf9a27; no voters:
I20260929 02:49:07.162760 6054 leader_election.cc:288] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:07.162896 6059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:07.163615 6059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [term 1 LEADER]: Becoming Leader. State: Replica: 9697cda8f1a846d1a60c89457cbf9a27, State: Running, Role: LEADER
I20260929 02:49:07.164121 6059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9697cda8f1a846d1a60c89457cbf9a27" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 39073 } }
I20260929 02:49:07.164744 6054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:07.166728 6060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9697cda8f1a846d1a60c89457cbf9a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9697cda8f1a846d1a60c89457cbf9a27" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 39073 } } }
I20260929 02:49:07.167186 6060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:07.166780 6061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9697cda8f1a846d1a60c89457cbf9a27. Latest consensus state: current_term: 1 leader_uuid: "9697cda8f1a846d1a60c89457cbf9a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9697cda8f1a846d1a60c89457cbf9a27" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 39073 } } }
I20260929 02:49:07.167414 6061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:07.168514 6067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:07.175786 6067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:07.183959 6067 catalog_manager.cc:1347] Generated new cluster ID: 36365f8195604dd89ef29f264a4f3a73
I20260929 02:49:07.184087 6067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:07.199209 6067 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:07.200439 6067 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:07.207906 6067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27: Generated new TSK 0
I20260929 02:49:07.208626 6067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:49:07.246447 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:39073
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:07.561144 6080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:07.561442 6080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:07.561581 6080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:07.568955 6080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:07.569088 6080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:07.569216 6080 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:49:07.569315 6080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:07.579223 6080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.197.190:39073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:07.580473 6080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:07.581771 6080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:07.594390 6085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:07.594390 6086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:07.594643 6088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:07.595072 6080 server_base.cc:1034] running on GCE node
I20260929 02:49:07.595633 6080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:07.596458 6080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:07.597658 6080 hybrid_clock.cc:648] HybridClock initialized: now 1790650147597599 us; error 34 us; skew 500 ppm
I20260929 02:49:07.597952 6080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:07 dist-test-slave-t6c1 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650147, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:07.601131 6080 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:49:07.602697 6080 webserver.cc:460] Webserver started at http://127.4.197.129:37959/ using document root <none> and password file <none>
I20260929 02:49:07.603503 6080 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:07.603603 6080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:07.603931 6080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:07.606796 6080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "cfe7bca2991a42a5a845653ceda16955"
format_stamp: "Formatted at 2026-09-29 02:49:07 on dist-test-slave-t6c1"
server_key: "3433bbf2369ff1d5890ab1440ff3ca1d"
server_key_iv: "b8242f50262b1d86446403160e7fce4e"
server_key_version: "encryptionkey@0"
I20260929 02:49:07.607568 6080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "cfe7bca2991a42a5a845653ceda16955"
format_stamp: "Formatted at 2026-09-29 02:49:07 on dist-test-slave-t6c1"
server_key: "3433bbf2369ff1d5890ab1440ff3ca1d"
server_key_iv: "b8242f50262b1d86446403160e7fce4e"
server_key_version: "encryptionkey@0"
I20260929 02:49:07.612529 6080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:49:07.615823 6095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:07.617101 6080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:49:07.617244 6080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "cfe7bca2991a42a5a845653ceda16955"
format_stamp: "Formatted at 2026-09-29 02:49:07 on dist-test-slave-t6c1"
server_key: "3433bbf2369ff1d5890ab1440ff3ca1d"
server_key_iv: "b8242f50262b1d86446403160e7fce4e"
server_key_version: "encryptionkey@0"
I20260929 02:49:07.617422 6080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:07.669730 6080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:07.673768 6080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:07.674031 6080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:07.676791 6080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:07.676887 6080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:07.677032 6080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:07.677101 6080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:49:07 dist-test-slave-t6c1 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650147, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:07.699283 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:07.677343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:41435 (local address 127.4.197.190:39073)
0929 02:49:07.679897 (+ 2554us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:07.679903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:07.679928 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:49:07.681398 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:07.681403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:07.681491 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:07.681587 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:07.683692 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:07.684826 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:07.687198 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:07.687767 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:07.687897 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:07.692742 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:07.692748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:07.692751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:07.692786 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:07.695306 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:07.696413 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:07.696417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:07.696419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:07.696552 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:07.697133 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:07.697137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:07.697139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:07.697519 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:07.697662 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:07.698570 (+ 908us) server_negotiation.cc:299] Negotiation successful
0929 02:49:07.698837 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2432,"thread_start_us":2335,"threads_started":1}
I20260929 02:49:07.700079 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:07.678125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:39073 (local address 127.4.197.129:41435)
0929 02:49:07.679104 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:07.679162 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:07.681122 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:07.681869 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:07.681881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:07.682622 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:07.683479 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:07.683514 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:07.685013 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:07.685018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:07.687055 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:07.687068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:07.688215 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:07.688220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:07.688547 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:07.689402 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:07.689433 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:07.692384 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:07.695495 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:07.695505 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:07.695529 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:07.696274 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:07.696669 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:07.696674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:07.696677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:07.697013 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:07.697740 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:07.697749 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:07.698169 (+ 420us) client_negotiation.cc:769] Sending connection context
0929 02:49:07.698407 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 02:49:07.698805 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":360,"threads_started":1}
I20260929 02:49:07.721882 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:07.709656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:44473 (local address 127.4.197.190:39073)
0929 02:49:07.709863 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:07.709869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:07.710338 (+ 469us) server_negotiation.cc:407] Connection header received
0929 02:49:07.710486 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:07.710492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:07.710571 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:07.710679 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:07.711752 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:07.712698 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:07.714562 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:07.714992 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:07.715102 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:07.716798 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:07.716804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:07.716807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:07.716844 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:07.719359 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:07.720209 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:07.720212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:07.720215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:07.720338 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:07.720682 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:07.720685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:07.720687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:07.720929 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:07.721055 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:07.721477 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 02:49:07.721651 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:49:07.722230 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:07.709675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:39073 (local address 127.4.197.129:44473)
0929 02:49:07.710250 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:07.710268 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:07.710387 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:07.710688 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:07.710691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:07.711132 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:07.711554 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:07.711571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:07.712836 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:07.712840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:07.714422 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:07.714438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:07.715184 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:07.715192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:07.715344 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:07.715942 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:07.715958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:07.716633 (+ 675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:07.719471 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:07.719476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:07.719480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:07.720098 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:07.720437 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:07.720441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:07.720446 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:07.720595 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:07.721040 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:07.721044 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:07.721252 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 02:49:07.721401 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 02:49:07.721731 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":303,"threads_started":1}
I20260929 02:49:07.733973 6080 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:35855
I20260929 02:49:07.734014 6215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:35855 every 8 connection(s)
I20260929 02:49:07.735733 6080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:07.737996 6216 heartbeater.cc:347] Connected to a master server at 127.4.197.190:39073
I20260929 02:49:07.738222 6216 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:07.738863 6216 heartbeater.cc:511] Master 127.4.197.190:39073 requested a full tablet report, sending...
I20260929 02:49:07.740736 6009 ts_manager.cc:194] Registered new tserver with Master: cfe7bca2991a42a5a845653ceda16955 (127.4.197.129:35855)
I20260929 02:49:07.742210 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:41435
I20260929 02:49:07.743464 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 6080
I20260929 02:49:07.743651 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:07.744230 4886 external_mini_cluster.cc:1428] Setting key 1e1991d81cb5dbffa3209b6e25d9e037
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:49:07.779742 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:39073
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:08.085223 6220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:08.085536 6220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:08.085637 6220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:08.092931 6220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:08.093066 6220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:08.093195 6220 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:49:08.093308 6220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:08.103560 6220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.197.190:39073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:08.104791 6220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:08.106077 6220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:08.118621 6226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:08.118710 6228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:08.118791 6225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:08.119407 6220 server_base.cc:1034] running on GCE node
I20260929 02:49:08.120119 6220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:08.121011 6220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:08.122227 6220 hybrid_clock.cc:648] HybridClock initialized: now 1790650148122144 us; error 63 us; skew 500 ppm
I20260929 02:49:08.122540 6220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:08 dist-test-slave-t6c1 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:08.126034 6220 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:49:08.128283 6220 webserver.cc:460] Webserver started at http://127.4.197.130:42367/ using document root <none> and password file <none>
I20260929 02:49:08.129019 6220 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:08.129135 6220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:08.129456 6220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:08.132597 6220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "9c666adb2f1640239ae99408ad2f85d6"
format_stamp: "Formatted at 2026-09-29 02:49:08 on dist-test-slave-t6c1"
server_key: "61fd8fc7d20f90e85d50123f36331253"
server_key_iv: "aafe8ac97abd31b56f129d150c7dcddc"
server_key_version: "encryptionkey@0"
I20260929 02:49:08.133430 6220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "9c666adb2f1640239ae99408ad2f85d6"
format_stamp: "Formatted at 2026-09-29 02:49:08 on dist-test-slave-t6c1"
server_key: "61fd8fc7d20f90e85d50123f36331253"
server_key_iv: "aafe8ac97abd31b56f129d150c7dcddc"
server_key_version: "encryptionkey@0"
I20260929 02:49:08.139065 6220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 02:49:08.142647 6235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:08.144151 6220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:08.144304 6220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "9c666adb2f1640239ae99408ad2f85d6"
format_stamp: "Formatted at 2026-09-29 02:49:08 on dist-test-slave-t6c1"
server_key: "61fd8fc7d20f90e85d50123f36331253"
server_key_iv: "aafe8ac97abd31b56f129d150c7dcddc"
server_key_version: "encryptionkey@0"
I20260929 02:49:08.144524 6220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:08.173353 6220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:08.178247 6220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:08.178526 6220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:08.181161 6220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:08.181265 6220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:08.181380 6220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:08.181443 6220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:49:08 dist-test-slave-t6c1 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:08.203971 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.182235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:39597 (local address 127.4.197.190:39073)
0929 02:49:08.182426 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.182432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.183674 (+ 1242us) server_negotiation.cc:407] Connection header received
0929 02:49:08.185925 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.185930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.186010 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.186088 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:08.188496 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.189475 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.192347 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.192910 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.193034 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.197209 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:08.197215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:08.197220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:08.197262 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:08.200514 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.201371 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.201377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.201380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.201521 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.202127 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.202132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.202135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.202446 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:08.202585 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.203425 (+ 840us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.203642 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:49:08.205349 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.182613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:39073 (local address 127.4.197.130:39597)
0929 02:49:08.183487 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.183543 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.185522 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.186621 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.186635 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.187435 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:08.188316 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.188346 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.189693 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.189699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.192164 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.192181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.193159 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.193166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.193417 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.194139 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:08.194163 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:08.196989 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:08.200671 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.200679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.200696 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.201189 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.201658 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.201663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.201670 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.202009 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.202581 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:08.202588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:08.202955 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.203312 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.203831 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":389,"threads_started":1}
I20260929 02:49:08.224241 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.210494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:39073 (local address 127.4.197.130:45035)
0929 02:49:08.211172 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.211188 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.211301 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.211585 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.211589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.212030 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:08.212297 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.212307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.213253 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.213258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.214240 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.214251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.215147 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.215154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.215366 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.216129 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:08.216147 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:08.217182 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:08.220545 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.220550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.220553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.221138 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.221635 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.221640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.221647 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.221817 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.222982 (+ 1165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:08.222987 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:08.223201 (+ 214us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.223357 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.223698 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":362,"threads_started":1}
I20260929 02:49:08.225481 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.210463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:45035 (local address 127.4.197.190:39073)
0929 02:49:08.210633 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.210639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.211262 (+ 623us) server_negotiation.cc:407] Connection header received
0929 02:49:08.211407 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.211413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.211476 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.211552 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:08.212431 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.213119 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.214381 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.214907 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.215058 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.217346 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:08.217352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:08.217357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:08.217404 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:08.220400 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.221311 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.221317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.221320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.221495 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.221966 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.221971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.221974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.222439 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:08.222626 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.224886 (+ 2260us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.225142 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:49:08.230621 6220 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:41655
I20260929 02:49:08.230657 6354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:41655 every 8 connection(s)
I20260929 02:49:08.231981 6220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:08.233923 6355 heartbeater.cc:347] Connected to a master server at 127.4.197.190:39073
I20260929 02:49:08.234125 6355 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:08.234741 6355 heartbeater.cc:511] Master 127.4.197.190:39073 requested a full tablet report, sending...
I20260929 02:49:08.235998 6009 ts_manager.cc:194] Registered new tserver with Master: 9c666adb2f1640239ae99408ad2f85d6 (127.4.197.130:41655)
I20260929 02:49:08.236742 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:39597
I20260929 02:49:08.237533 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 6220
I20260929 02:49:08.237731 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:08.238294 4886 external_mini_cluster.cc:1428] Setting key 4bd7a5edf825bac2777a38151c193879
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:49:08.272444 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:39073
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:08.579936 6359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:08.580251 6359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:08.580379 6359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:08.588136 6359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:08.588289 6359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:08.588418 6359 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:49:08.588528 6359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:08.598495 6359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.197.190:39073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:08.599840 6359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:08.601171 6359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:08.613222 6364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:08.613288 6365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:08.613324 6367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:08.614181 6359 server_base.cc:1034] running on GCE node
I20260929 02:49:08.614698 6359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:08.615568 6359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:08.616766 6359 hybrid_clock.cc:648] HybridClock initialized: now 1790650148616711 us; error 32 us; skew 500 ppm
I20260929 02:49:08.617038 6359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:08 dist-test-slave-t6c1 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:08.626480 6359 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:49:08.627928 6359 webserver.cc:460] Webserver started at http://127.4.197.131:46075/ using document root <none> and password file <none>
I20260929 02:49:08.628579 6359 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:08.628662 6359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:08.628935 6359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:08.631824 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "08b5bfd24bdb4f10a398777bb78e43ce"
format_stamp: "Formatted at 2026-09-29 02:49:08 on dist-test-slave-t6c1"
server_key: "4ebf28a7cf85f074c30d72b2843a3df3"
server_key_iv: "78d7fd9c2f3d2f48d6ba67e63a252c8f"
server_key_version: "encryptionkey@0"
I20260929 02:49:08.632563 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "08b5bfd24bdb4f10a398777bb78e43ce"
format_stamp: "Formatted at 2026-09-29 02:49:08 on dist-test-slave-t6c1"
server_key: "4ebf28a7cf85f074c30d72b2843a3df3"
server_key_iv: "78d7fd9c2f3d2f48d6ba67e63a252c8f"
server_key_version: "encryptionkey@0"
I20260929 02:49:08.637713 6359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:49:08.641433 6374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:08.643000 6359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:49:08.643172 6359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "08b5bfd24bdb4f10a398777bb78e43ce"
format_stamp: "Formatted at 2026-09-29 02:49:08 on dist-test-slave-t6c1"
server_key: "4ebf28a7cf85f074c30d72b2843a3df3"
server_key_iv: "78d7fd9c2f3d2f48d6ba67e63a252c8f"
server_key_version: "encryptionkey@0"
I20260929 02:49:08.643410 6359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:08.659432 6359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:08.663741 6359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:08.664008 6359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:08.666707 6359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:08.666817 6359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:08.666958 6359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:08.667037 6359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 02:49:08 dist-test-slave-t6c1 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:08.687628 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.666982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:43201 (local address 127.4.197.190:39073)
0929 02:49:08.667161 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.667167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.668520 (+ 1353us) server_negotiation.cc:407] Connection header received
0929 02:49:08.670558 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.670563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.670648 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.670735 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:08.672613 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.673599 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.675880 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.676385 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.676503 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.680396 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:08.680405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:08.680412 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:08.680465 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:08.683820 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.684936 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.684942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.684945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.685099 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.685622 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.685627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.685631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.685897 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:08.686039 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.687051 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.687279 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:49:08.688611 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.667504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:39073 (local address 127.4.197.131:43201)
0929 02:49:08.668345 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.668393 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.670283 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.671027 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.671035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.671609 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:08.672433 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.672458 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.673791 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.673796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.675739 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.675751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.676593 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.676600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.676788 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.677401 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:08.677445 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:08.680120 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:08.683975 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.683986 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.684008 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.684771 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.685212 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.685220 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.685224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.685508 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.686122 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:08.686130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:08.686580 (+ 450us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.686896 (+ 316us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.687463 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":332,"threads_started":1}
I20260929 02:49:08.713871 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.700035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:54139 (local address 127.4.197.190:39073)
0929 02:49:08.700219 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.700224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.700830 (+ 606us) server_negotiation.cc:407] Connection header received
0929 02:49:08.701086 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.701092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.701153 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.701226 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:08.702722 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.703663 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.705438 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.705846 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.705969 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.708520 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:08.708525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:08.708528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:08.708559 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:08.711072 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.711877 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.711883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.711887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.712051 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.712432 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.712437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.712441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.712683 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:08.712807 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.713297 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.713516 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:49:08.713961 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.700043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:39073 (local address 127.4.197.131:54139)
0929 02:49:08.700716 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.700739 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.700922 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.701297 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.701303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.701939 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:08.702552 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.702570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.703801 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.703805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.705297 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.705310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.706023 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.706029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.706184 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.706888 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:08.706949 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:08.708325 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:08.711198 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.711203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.711207 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.711772 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.712185 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.712191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.712195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.712319 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.712791 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:08.712795 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:08.713034 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.713193 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.713516 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":342,"threads_started":1}
I20260929 02:49:08.722123 6359 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:41947
I20260929 02:49:08.722141 6493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:41947 every 8 connection(s)
I20260929 02:49:08.723524 6359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:08.725625 6494 heartbeater.cc:347] Connected to a master server at 127.4.197.190:39073
I20260929 02:49:08.725821 6494 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:08.726467 6494 heartbeater.cc:511] Master 127.4.197.190:39073 requested a full tablet report, sending...
I20260929 02:49:08.727872 6009 ts_manager.cc:194] Registered new tserver with Master: 08b5bfd24bdb4f10a398777bb78e43ce (127.4.197.131:41947)
I20260929 02:49:08.728091 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 6359
I20260929 02:49:08.728258 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:08.728657 4886 external_mini_cluster.cc:1428] Setting key 6495028de5afda5ee9275898ae1017d9
I20260929 02:49:08.728761 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:43201
I20260929 02:49:08.735172 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:08.745093 6216 heartbeater.cc:503] Master 127.4.197.190:39073 was elected leader, sending a full tablet report...
Sep 29 02:49:08 dist-test-slave-t6c1 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:49:08 dist-test-slave-t6c1 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:08.777853 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.764009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.4.197.190:39073)
0929 02:49:08.764272 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.764279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.764507 (+ 228us) server_negotiation.cc:407] Connection header received
0929 02:49:08.764677 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.764682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.764758 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.765055 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:08.766425 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.767311 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.768597 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.769111 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.769231 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.772872 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:08.772877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:08.772880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:08.772916 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:08.775543 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.776156 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.776160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.776162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.776299 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.776615 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.776618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.776620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.776925 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:08.777103 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.777317 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.777486 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 02:49:08.781685 6009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:08.788255 6009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:08.810174 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.799534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.4.197.130:41655)
0929 02:49:08.803194 (+ 3660us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.803201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.803224 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:49:08.803302 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.803307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.803501 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.804279 (+ 778us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:08.804466 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.805899 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.808066 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.809085 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.809187 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.809329 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.809717 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.809890 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3505,"thread_start_us":3357,"threads_started":1}
I20260929 02:49:08.810209 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.799278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:41655 (local address 127.0.0.1:33944)
0929 02:49:08.800279 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.800328 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.800492 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.803697 (+ 3205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.803702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.803728 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:08.804100 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.804112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.806038 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.806044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.807902 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.807916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.809247 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.809253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.809436 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.809491 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.809609 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.809790 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":22,"thread_start_us":579,"threads_started":1}
I20260929 02:49:08.810248 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.800613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:35855 (local address 127.0.0.1:54642)
0929 02:49:08.801576 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.801592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.801678 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.802851 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.802859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.802891 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:08.803218 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.803228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.805068 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.805073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.806829 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.806849 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.809513 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.809520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.809680 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.809709 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.809787 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.809900 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":722,"threads_started":1}
I20260929 02:49:08.811038 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.799740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:41947 (local address 127.0.0.1:37480)
0929 02:49:08.801012 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.801028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.801131 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.803542 (+ 2411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.803547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.803570 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:08.804069 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.804085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.806153 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.806157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.808200 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.808213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.810172 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.810179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.810603 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.810633 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.810682 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.810758 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":49,"thread_start_us":1018,"threads_started":1}
I20260929 02:49:08.811123 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.801121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.4.197.129:35855)
0929 02:49:08.802119 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.802129 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.802156 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:49:08.802261 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.802313 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.802610 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.803387 (+ 777us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:08.803567 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.804920 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.807032 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.808483 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.809434 (+ 951us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.809617 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.810394 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.810843 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":1052,"threads_started":1}
I20260929 02:49:08.812028 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.800324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.4.197.131:41947)
0929 02:49:08.802989 (+ 2665us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.802997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.803029 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:49:08.803121 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.803127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.803353 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.803542 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:08.804429 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.805995 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.808619 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.809954 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.811329 (+ 1375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.811531 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.811592 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.811732 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2467,"thread_start_us":2264,"threads_started":1}
I20260929 02:49:08.814052 6146 tablet_service.cc:1467] Processing CreateTablet for tablet a9257301b01841b0b0d6b4c0de54ad25 (DEFAULT_TABLE table=test-table [id=b2ebe175d55a4fc487dcdca3693aadcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:08.814082 6285 tablet_service.cc:1467] Processing CreateTablet for tablet a9257301b01841b0b0d6b4c0de54ad25 (DEFAULT_TABLE table=test-table [id=b2ebe175d55a4fc487dcdca3693aadcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:08.815207 6146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9257301b01841b0b0d6b4c0de54ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:08.815362 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9257301b01841b0b0d6b4c0de54ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:08.814994 6424 tablet_service.cc:1467] Processing CreateTablet for tablet a9257301b01841b0b0d6b4c0de54ad25 (DEFAULT_TABLE table=test-table [id=b2ebe175d55a4fc487dcdca3693aadcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:08.816157 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9257301b01841b0b0d6b4c0de54ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:08.825909 6513 tablet_bootstrap.cc:492] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: Bootstrap starting.
I20260929 02:49:08.829547 6515 tablet_bootstrap.cc:492] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: Bootstrap starting.
I20260929 02:49:08.829239 6513 tablet_bootstrap.cc:654] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:08.831164 6514 tablet_bootstrap.cc:492] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: Bootstrap starting.
I20260929 02:49:08.832547 6513 log.cc:824] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:08.833559 6515 tablet_bootstrap.cc:654] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:08.834522 6514 tablet_bootstrap.cc:654] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:08.834729 6515 log.cc:824] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:08.835541 6514 log.cc:824] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:08.835858 6513 tablet_bootstrap.cc:492] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: No bootstrap required, opened a new log
I20260929 02:49:08.836133 6513 ts_tablet_manager.cc:1397] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 02:49:08.837716 6515 tablet_bootstrap.cc:492] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: No bootstrap required, opened a new log
I20260929 02:49:08.837972 6515 ts_tablet_manager.cc:1397] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 02:49:08.844980 6514 tablet_bootstrap.cc:492] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: No bootstrap required, opened a new log
I20260929 02:49:08.845219 6514 ts_tablet_manager.cc:1397] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20260929 02:49:08.847102 6513 raft_consensus.cc:348] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } }
I20260929 02:49:08.847589 6513 raft_consensus.cc:374] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:08.847719 6513 raft_consensus.cc:729] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe7bca2991a42a5a845653ceda16955, State: Initialized, Role: FOLLOWER
I20260929 02:49:08.848378 6513 consensus_queue.cc:260] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } }
I20260929 02:49:08.848791 6515 raft_consensus.cc:348] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } }
I20260929 02:49:08.849331 6515 raft_consensus.cc:374] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:08.849422 6515 raft_consensus.cc:729] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b5bfd24bdb4f10a398777bb78e43ce, State: Initialized, Role: FOLLOWER
I20260929 02:49:08.850035 6515 consensus_queue.cc:260] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } }
I20260929 02:49:08.851017 6494 heartbeater.cc:503] Master 127.4.197.190:39073 was elected leader, sending a full tablet report...
I20260929 02:49:08.852192 6513 ts_tablet_manager.cc:1428] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260929 02:49:08.853075 6515 ts_tablet_manager.cc:1428] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260929 02:49:08.853581 6514 raft_consensus.cc:348] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } }
I20260929 02:49:08.853972 6514 raft_consensus.cc:374] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:08.854069 6514 raft_consensus.cc:729] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c666adb2f1640239ae99408ad2f85d6, State: Initialized, Role: FOLLOWER
I20260929 02:49:08.854579 6514 consensus_queue.cc:260] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } }
I20260929 02:49:08.855717 6355 heartbeater.cc:503] Master 127.4.197.190:39073 was elected leader, sending a full tablet report...
I20260929 02:49:08.856338 6514 ts_tablet_manager.cc:1428] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260929 02:49:08.884598 6519 raft_consensus.cc:482] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:08.884881 6519 raft_consensus.cc:504] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } }
I20260929 02:49:08.886533 6519 leader_election.cc:288] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c666adb2f1640239ae99408ad2f85d6 (127.4.197.130:41655), 08b5bfd24bdb4f10a398777bb78e43ce (127.4.197.131:41947)
I20260929 02:49:08.893446 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.887023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:53441 (local address 127.4.197.131:41947)
0929 02:49:08.887266 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.887273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.888017 (+ 744us) server_negotiation.cc:407] Connection header received
0929 02:49:08.888189 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.888195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.888268 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.888350 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:08.888980 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.889735 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.891771 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.892514 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.892644 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.892718 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.893067 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.893195 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:49:08.893445 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.887577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:56393 (local address 127.4.197.130:41655)
0929 02:49:08.887733 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.887737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.888225 (+ 488us) server_negotiation.cc:407] Connection header received
0929 02:49:08.888366 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.888371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.888450 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.888555 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:08.889106 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.889913 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.891756 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.892549 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.892715 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.892797 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.893063 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.893199 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 02:49:08.893572 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.886764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:41655 (local address 127.4.197.129:56393)
0929 02:49:08.888105 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.888123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.888241 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.888577 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.888581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.888603 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:08.888950 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.888962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.890048 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.890052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.891602 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.891617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.892769 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.892774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.892915 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.892939 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.893020 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.893175 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"mutex_wait_us":10,"thread_start_us":1010,"threads_started":1}
I20260929 02:49:08.894016 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.886753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:41947 (local address 127.4.197.129:53441)
0929 02:49:08.887906 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.887923 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.888043 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.888405 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.888410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.888436 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:08.888830 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.888842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.889854 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.889859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.891625 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.891637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.892726 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.892733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.892920 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.892943 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.893022 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.893199 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":90,"thread_start_us":704,"threads_started":1}
I20260929 02:49:08.894560 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9257301b01841b0b0d6b4c0de54ad25" candidate_uuid: "cfe7bca2991a42a5a845653ceda16955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c666adb2f1640239ae99408ad2f85d6" is_pre_election: true
I20260929 02:49:08.894963 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9257301b01841b0b0d6b4c0de54ad25" candidate_uuid: "cfe7bca2991a42a5a845653ceda16955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" is_pre_election: true
I20260929 02:49:08.895190 6305 raft_consensus.cc:2454] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfe7bca2991a42a5a845653ceda16955 in term 0.
I20260929 02:49:08.895512 6446 raft_consensus.cc:2454] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfe7bca2991a42a5a845653ceda16955 in term 0.
I20260929 02:49:08.895908 6099 leader_election.cc:302] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c666adb2f1640239ae99408ad2f85d6, cfe7bca2991a42a5a845653ceda16955; no voters:
I20260929 02:49:08.896340 6519 raft_consensus.cc:2781] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:08.896472 6519 raft_consensus.cc:482] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:08.896566 6519 raft_consensus.cc:3037] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:08.899263 6519 raft_consensus.cc:504] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } }
I20260929 02:49:08.900081 6519 leader_election.cc:288] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [CANDIDATE]: Term 1 election: Requested vote from peers 9c666adb2f1640239ae99408ad2f85d6 (127.4.197.130:41655), 08b5bfd24bdb4f10a398777bb78e43ce (127.4.197.131:41947)
I20260929 02:49:08.900310 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9257301b01841b0b0d6b4c0de54ad25" candidate_uuid: "cfe7bca2991a42a5a845653ceda16955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c666adb2f1640239ae99408ad2f85d6"
I20260929 02:49:08.900367 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9257301b01841b0b0d6b4c0de54ad25" candidate_uuid: "cfe7bca2991a42a5a845653ceda16955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b5bfd24bdb4f10a398777bb78e43ce"
I20260929 02:49:08.900548 6446 raft_consensus.cc:3037] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:08.900543 6305 raft_consensus.cc:3037] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:08.902742 6305 raft_consensus.cc:2454] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfe7bca2991a42a5a845653ceda16955 in term 1.
I20260929 02:49:08.902758 6446 raft_consensus.cc:2454] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfe7bca2991a42a5a845653ceda16955 in term 1.
I20260929 02:49:08.903112 6099 leader_election.cc:302] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c666adb2f1640239ae99408ad2f85d6, cfe7bca2991a42a5a845653ceda16955; no voters:
I20260929 02:49:08.903389 6519 raft_consensus.cc:2781] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:08.903766 6519 raft_consensus.cc:686] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 1 LEADER]: Becoming Leader. State: Replica: cfe7bca2991a42a5a845653ceda16955, State: Running, Role: LEADER
I20260929 02:49:08.904177 6519 consensus_queue.cc:237] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } }
I20260929 02:49:08.908566 6009 catalog_manager.cc:5495] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfe7bca2991a42a5a845653ceda16955 (127.4.197.129). New cstate: current_term: 1 leader_uuid: "cfe7bca2991a42a5a845653ceda16955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } health_report { overall_health: HEALTHY } } }
I20260929 02:49:08.940102 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.934818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.4.197.129:35855)
0929 02:49:08.935025 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.935030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.935164 (+ 134us) server_negotiation.cc:407] Connection header received
0929 02:49:08.935417 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.935421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.935494 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.935638 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:08.936007 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.937079 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.938313 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.938688 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.938770 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.939215 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:08.939692 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:08.939796 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.939849 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.939911 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:49:08.947671 6446 raft_consensus.cc:1261] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 FOLLOWER]: Refusing update from remote peer cfe7bca2991a42a5a845653ceda16955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:08.947671 6305 raft_consensus.cc:1261] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6 [term 1 FOLLOWER]: Refusing update from remote peer cfe7bca2991a42a5a845653ceda16955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:08.948675 6524 consensus_queue.cc:1035] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:08.949007 6519 consensus_queue.cc:1035] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:08.969241 6528 mvcc.cc:205] Tried to move back new op lower bound from 7334503010079698944 to 7334503009914888192. Current Snapshot: MvccSnapshot[applied={T|T < 7334503010079698944}]
I20260929 02:49:08.982590 6009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:08.982825 6009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:08.986744 6009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27: Sending DeleteTablet for 3 replicas of tablet a9257301b01841b0b0d6b4c0de54ad25
I20260929 02:49:08.988144 6146 tablet_service.cc:1514] Processing DeleteTablet for tablet a9257301b01841b0b0d6b4c0de54ad25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:08 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:54642
I20260929 02:49:08.988131 6285 tablet_service.cc:1514] Processing DeleteTablet for tablet a9257301b01841b0b0d6b4c0de54ad25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:08 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:33944
I20260929 02:49:08.988541 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet a9257301b01841b0b0d6b4c0de54ad25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:08 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:37480
I20260929 02:49:08.995343 6009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:08.996284 6009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:09.000475 6539 tablet_replica.cc:330] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: stopping tablet replica
I20260929 02:49:09.000808 6537 tablet_replica.cc:330] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: stopping tablet replica
I20260929 02:49:09.001049 6539 raft_consensus.cc:2229] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:09.001372 6537 raft_consensus.cc:2229] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:09.001592 6539 raft_consensus.cc:2258] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:09.002019 6537 raft_consensus.cc:2258] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:09.002871 6538 tablet_replica.cc:330] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: stopping tablet replica
I20260929 02:49:09.003360 6538 raft_consensus.cc:2229] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:09.004508 6538 raft_consensus.cc:2258] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:09.004550 6539 ts_tablet_manager.cc:1905] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:09.004550 6537 ts_tablet_manager.cc:1905] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:09.008821 6538 ts_tablet_manager.cc:1905] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:09.010895 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcf80a981c0480db2557c89c2977ec3 (DEFAULT_TABLE table=test-table [id=33e94b71009b48c3914b3063b825a8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:09.011428 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcf80a981c0480db2557c89c2977ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:09.011675 6539 ts_tablet_manager.cc:1918] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:09.011847 6539 log.cc:1196] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/a9257301b01841b0b0d6b4c0de54ad25
I20260929 02:49:09.012468 6539 ts_tablet_manager.cc:1939] T a9257301b01841b0b0d6b4c0de54ad25 P 08b5bfd24bdb4f10a398777bb78e43ce: Deleting consensus metadata
I20260929 02:49:09.014782 5994 catalog_manager.cc:4841] TS 08b5bfd24bdb4f10a398777bb78e43ce (127.4.197.131:41947): tablet a9257301b01841b0b0d6b4c0de54ad25 (table test-table [id=b2ebe175d55a4fc487dcdca3693aadcd]) successfully deleted
I20260929 02:49:09.016319 6146 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcf80a981c0480db2557c89c2977ec3 (DEFAULT_TABLE table=test-table [id=33e94b71009b48c3914b3063b825a8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:09.017484 6146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcf80a981c0480db2557c89c2977ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:09.017645 6538 ts_tablet_manager.cc:1918] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:09.017844 6538 log.cc:1196] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/a9257301b01841b0b0d6b4c0de54ad25
I20260929 02:49:09.021303 6538 ts_tablet_manager.cc:1939] T a9257301b01841b0b0d6b4c0de54ad25 P cfe7bca2991a42a5a845653ceda16955: Deleting consensus metadata
I20260929 02:49:09.021953 6515 tablet_bootstrap.cc:492] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: Bootstrap starting.
I20260929 02:49:09.022361 6537 ts_tablet_manager.cc:1918] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:09.022622 6537 log.cc:1196] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/a9257301b01841b0b0d6b4c0de54ad25
I20260929 02:49:09.023231 6537 ts_tablet_manager.cc:1939] T a9257301b01841b0b0d6b4c0de54ad25 P 9c666adb2f1640239ae99408ad2f85d6: Deleting consensus metadata
I20260929 02:49:09.023231 5996 catalog_manager.cc:4841] TS cfe7bca2991a42a5a845653ceda16955 (127.4.197.129:35855): tablet a9257301b01841b0b0d6b4c0de54ad25 (table test-table [id=b2ebe175d55a4fc487dcdca3693aadcd]) successfully deleted
I20260929 02:49:09.022485 6285 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcf80a981c0480db2557c89c2977ec3 (DEFAULT_TABLE table=test-table [id=33e94b71009b48c3914b3063b825a8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:09.023792 6515 tablet_bootstrap.cc:654] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:09.023816 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcf80a981c0480db2557c89c2977ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:09.025329 5996 catalog_manager.cc:4841] TS 9c666adb2f1640239ae99408ad2f85d6 (127.4.197.130:41655): tablet a9257301b01841b0b0d6b4c0de54ad25 (table test-table [id=b2ebe175d55a4fc487dcdca3693aadcd]) successfully deleted
I20260929 02:49:09.027657 6513 tablet_bootstrap.cc:492] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: Bootstrap starting.
I20260929 02:49:09.029315 6513 tablet_bootstrap.cc:654] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:09.030855 6514 tablet_bootstrap.cc:492] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: Bootstrap starting.
I20260929 02:49:09.032841 6514 tablet_bootstrap.cc:654] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:09.039978 6513 tablet_bootstrap.cc:492] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: No bootstrap required, opened a new log
I20260929 02:49:09.040238 6513 ts_tablet_manager.cc:1397] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260929 02:49:09.040457 6515 tablet_bootstrap.cc:492] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: No bootstrap required, opened a new log
I20260929 02:49:09.040648 6515 ts_tablet_manager.cc:1397] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20260929 02:49:09.041330 6513 raft_consensus.cc:348] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.041666 6513 raft_consensus.cc:374] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:09.041590 6515 raft_consensus.cc:348] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.041864 6515 raft_consensus.cc:374] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:09.041805 6513 raft_consensus.cc:729] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe7bca2991a42a5a845653ceda16955, State: Initialized, Role: FOLLOWER
I20260929 02:49:09.041970 6515 raft_consensus.cc:729] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b5bfd24bdb4f10a398777bb78e43ce, State: Initialized, Role: FOLLOWER
I20260929 02:49:09.042255 6513 consensus_queue.cc:260] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.043958 6513 ts_tablet_manager.cc:1428] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:49:09.044006 6514 tablet_bootstrap.cc:492] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: No bootstrap required, opened a new log
I20260929 02:49:09.044186 6514 ts_tablet_manager.cc:1397] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260929 02:49:09.044435 6515 consensus_queue.cc:260] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.045239 6515 ts_tablet_manager.cc:1428] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260929 02:49:09.045754 6514 raft_consensus.cc:348] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.046106 6514 raft_consensus.cc:374] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:09.046296 6514 raft_consensus.cc:729] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c666adb2f1640239ae99408ad2f85d6, State: Initialized, Role: FOLLOWER
I20260929 02:49:09.046708 6514 consensus_queue.cc:260] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.047734 6514 ts_tablet_manager.cc:1428] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 02:49:09.128481 6520 raft_consensus.cc:482] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:09.129202 6520 raft_consensus.cc:504] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.131116 6520 leader_election.cc:288] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfe7bca2991a42a5a845653ceda16955 (127.4.197.129:35855), 9c666adb2f1640239ae99408ad2f85d6 (127.4.197.130:41655)
I20260929 02:49:09.139370 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.131347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:35855 (local address 127.4.197.131:59825)
0929 02:49:09.132083 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:09.132102 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:09.132233 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:09.132768 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:09.132773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:09.132799 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:09.133007 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.133018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.134539 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.134545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.136594 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.136610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.138527 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.138533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.138758 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.138796 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:49:09.138884 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:49:09.139068 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615}
I20260929 02:49:09.140180 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.131519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:59825 (local address 127.4.197.129:35855)
0929 02:49:09.131936 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.131942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.132338 (+ 396us) server_negotiation.cc:407] Connection header received
0929 02:49:09.132413 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.132419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.132509 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.132609 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:09.133162 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.134350 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.136778 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.138181 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.138347 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.138428 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.139702 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.139845 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260929 02:49:09.145110 6168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcf80a981c0480db2557c89c2977ec3" candidate_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe7bca2991a42a5a845653ceda16955" is_pre_election: true
I20260929 02:49:09.145615 6168 raft_consensus.cc:2454] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08b5bfd24bdb4f10a398777bb78e43ce in term 0.
I20260929 02:49:09.146263 6378 leader_election.cc:302] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b5bfd24bdb4f10a398777bb78e43ce, cfe7bca2991a42a5a845653ceda16955; no voters:
I20260929 02:49:09.146698 6520 raft_consensus.cc:2781] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:09.146828 6520 raft_consensus.cc:482] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:09.146956 6520 raft_consensus.cc:3037] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:09.147547 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.132081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:54125 (local address 127.4.197.130:41655)
0929 02:49:09.132255 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.132261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.140691 (+ 8430us) server_negotiation.cc:407] Connection header received
0929 02:49:09.141166 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.141173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.141266 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.141326 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:09.141876 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.143034 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.145184 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.146437 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.146601 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.146677 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.147067 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.147270 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:49:09.148355 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.131673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:41655 (local address 127.4.197.131:54125)
0929 02:49:09.140583 (+ 8910us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:09.140605 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:09.140989 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:09.141510 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:09.141515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:09.141543 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:09.141715 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.141725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.143211 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.143216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.144988 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.145005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.146625 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.146631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.146813 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.146854 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 02:49:09.147801 (+ 947us) client_negotiation.cc:240] Negotiation successful
0929 02:49:09.147969 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8802,"thread_start_us":12021,"threads_started":1}
I20260929 02:49:09.149703 6520 raft_consensus.cc:504] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.150336 6520 leader_election.cc:288] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [CANDIDATE]: Term 1 election: Requested vote from peers cfe7bca2991a42a5a845653ceda16955 (127.4.197.129:35855), 9c666adb2f1640239ae99408ad2f85d6 (127.4.197.130:41655)
I20260929 02:49:09.151077 6168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcf80a981c0480db2557c89c2977ec3" candidate_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe7bca2991a42a5a845653ceda16955"
I20260929 02:49:09.151296 6168 raft_consensus.cc:3037] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:09.151494 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcf80a981c0480db2557c89c2977ec3" candidate_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c666adb2f1640239ae99408ad2f85d6" is_pre_election: true
I20260929 02:49:09.151759 6305 raft_consensus.cc:2454] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08b5bfd24bdb4f10a398777bb78e43ce in term 0.
I20260929 02:49:09.151739 6304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcf80a981c0480db2557c89c2977ec3" candidate_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c666adb2f1640239ae99408ad2f85d6"
I20260929 02:49:09.152129 6304 raft_consensus.cc:3037] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:09.154587 6168 raft_consensus.cc:2454] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08b5bfd24bdb4f10a398777bb78e43ce in term 1.
I20260929 02:49:09.154995 6378 leader_election.cc:302] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b5bfd24bdb4f10a398777bb78e43ce, cfe7bca2991a42a5a845653ceda16955; no voters:
I20260929 02:49:09.155529 6304 raft_consensus.cc:2454] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08b5bfd24bdb4f10a398777bb78e43ce in term 1.
I20260929 02:49:09.155794 6520 raft_consensus.cc:2781] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:09.156215 6520 raft_consensus.cc:686] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 LEADER]: Becoming Leader. State: Replica: 08b5bfd24bdb4f10a398777bb78e43ce, State: Running, Role: LEADER
I20260929 02:49:09.156742 6520 consensus_queue.cc:237] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.160168 6009 catalog_manager.cc:5495] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 08b5bfd24bdb4f10a398777bb78e43ce (127.4.197.131). New cstate: current_term: 1 leader_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } health_report { overall_health: HEALTHY } } }
Sep 29 02:49:09 dist-test-slave-t6c1 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650147, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:09.219234 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.205209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:39073 (local address 127.0.0.1:57674)
0929 02:49:09.205781 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:09.205798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:09.205920 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:09.206264 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:09.206267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:09.206952 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:09.207199 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.207208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.208603 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.208606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.209514 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.209524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.210450 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.210454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.210569 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.211226 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:09.211242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:09.213421 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:09.217020 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:09.217030 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:09.217035 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:09.217422 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:09.217780 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:09.217784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:09.217786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:09.217918 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:09.218435 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:09.218440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:09.218570 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 02:49:09.218693 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 02:49:09.218875 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":311,"threads_started":1}
I20260929 02:49:09.219451 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.205152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.4.197.190:39073)
0929 02:49:09.205333 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.205339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.205890 (+ 551us) server_negotiation.cc:407] Connection header received
0929 02:49:09.206025 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.206030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.206128 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.206243 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:09.207342 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.208494 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.209645 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.210280 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.210402 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.213585 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:09.213591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:09.213596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:09.213630 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:09.216885 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.217522 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.217527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.217530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.217685 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.218000 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.218005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.218009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.218310 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:09.218436 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.218799 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.219037 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:49:09.224706 6009 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:57674:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:49:09.225713 6009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260929 02:49:09.225979 6495 tablet.cc:2369] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:09.232820 6424 tablet_service.cc:1467] Processing CreateTablet for tablet a5e6c823c75643a4828d4687304041cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=addf313cb532416fa42c161641ab7115]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:49:09.233316 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e6c823c75643a4828d4687304041cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 02:49:09.234833 6356 tablet.cc:2369] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:09.235471 6146 tablet_service.cc:1467] Processing CreateTablet for tablet a5e6c823c75643a4828d4687304041cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=addf313cb532416fa42c161641ab7115]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:49:09.236049 6146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e6c823c75643a4828d4687304041cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:09.238391 6515 tablet_bootstrap.cc:492] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: Bootstrap starting.
I20260929 02:49:09.238965 6285 tablet_service.cc:1467] Processing CreateTablet for tablet a5e6c823c75643a4828d4687304041cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=addf313cb532416fa42c161641ab7115]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:49:09.239506 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e6c823c75643a4828d4687304041cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 02:49:09.240147 6217 tablet.cc:2369] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:09.240988 6515 tablet_bootstrap.cc:654] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:09.248308 6514 tablet_bootstrap.cc:492] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: Bootstrap starting.
I20260929 02:49:09.250000 6514 tablet_bootstrap.cc:654] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:09.250360 6515 tablet_bootstrap.cc:492] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: No bootstrap required, opened a new log
I20260929 02:49:09.250598 6515 ts_tablet_manager.cc:1397] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260929 02:49:09.251557 6513 tablet_bootstrap.cc:492] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: Bootstrap starting.
I20260929 02:49:09.251509 6515 raft_consensus.cc:348] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.251876 6515 raft_consensus.cc:374] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:09.251984 6515 raft_consensus.cc:729] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b5bfd24bdb4f10a398777bb78e43ce, State: Initialized, Role: FOLLOWER
I20260929 02:49:09.252269 6515 consensus_queue.cc:260] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.252720 6520 tablet_replica.cc:439] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } }
I20260929 02:49:09.252976 6520 tablet_replica.cc:442] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:09.253360 6513 tablet_bootstrap.cc:654] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:09.255863 6515 ts_tablet_manager.cc:1428] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 02:49:09.266196 6513 tablet_bootstrap.cc:492] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: No bootstrap required, opened a new log
I20260929 02:49:09.266419 6513 ts_tablet_manager.cc:1397] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.008s
I20260929 02:49:09.267283 6513 raft_consensus.cc:348] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.267562 6513 raft_consensus.cc:374] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:09.267656 6513 raft_consensus.cc:729] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe7bca2991a42a5a845653ceda16955, State: Initialized, Role: FOLLOWER
I20260929 02:49:09.268010 6513 consensus_queue.cc:260] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.268474 6524 tablet_replica.cc:439] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } }
I20260929 02:49:09.268821 6524 tablet_replica.cc:442] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:09.269654 6513 ts_tablet_manager.cc:1428] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:09.270481 6514 tablet_bootstrap.cc:492] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: No bootstrap required, opened a new log
I20260929 02:49:09.270664 6514 ts_tablet_manager.cc:1397] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.003s
I20260929 02:49:09.271360 6514 raft_consensus.cc:348] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.271543 6514 raft_consensus.cc:374] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:09.271615 6514 raft_consensus.cc:729] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c666adb2f1640239ae99408ad2f85d6, State: Initialized, Role: FOLLOWER
I20260929 02:49:09.271854 6514 consensus_queue.cc:260] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.272382 6514 ts_tablet_manager.cc:1428] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:49:09.272311 6521 tablet_replica.cc:439] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } }
I20260929 02:49:09.272714 6521 tablet_replica.cc:442] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:09.427098 6524 raft_consensus.cc:482] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:09.427292 6524 raft_consensus.cc:504] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.427821 6524 leader_election.cc:288] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c666adb2f1640239ae99408ad2f85d6 (127.4.197.130:41655), 08b5bfd24bdb4f10a398777bb78e43ce (127.4.197.131:41947)
I20260929 02:49:09.428398 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e6c823c75643a4828d4687304041cf" candidate_uuid: "cfe7bca2991a42a5a845653ceda16955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" is_pre_election: true
I20260929 02:49:09.428380 6304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e6c823c75643a4828d4687304041cf" candidate_uuid: "cfe7bca2991a42a5a845653ceda16955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c666adb2f1640239ae99408ad2f85d6" is_pre_election: true
I20260929 02:49:09.428635 6446 raft_consensus.cc:2454] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfe7bca2991a42a5a845653ceda16955 in term 0.
I20260929 02:49:09.428633 6304 raft_consensus.cc:2454] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfe7bca2991a42a5a845653ceda16955 in term 0.
I20260929 02:49:09.429039 6097 leader_election.cc:302] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b5bfd24bdb4f10a398777bb78e43ce, cfe7bca2991a42a5a845653ceda16955; no voters:
I20260929 02:49:09.429313 6524 raft_consensus.cc:2781] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:09.429438 6524 raft_consensus.cc:482] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:09.429518 6524 raft_consensus.cc:3037] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:09.431485 6524 raft_consensus.cc:504] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.432005 6524 leader_election.cc:288] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [CANDIDATE]: Term 1 election: Requested vote from peers 9c666adb2f1640239ae99408ad2f85d6 (127.4.197.130:41655), 08b5bfd24bdb4f10a398777bb78e43ce (127.4.197.131:41947)
I20260929 02:49:09.432375 6304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e6c823c75643a4828d4687304041cf" candidate_uuid: "cfe7bca2991a42a5a845653ceda16955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c666adb2f1640239ae99408ad2f85d6"
I20260929 02:49:09.432439 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e6c823c75643a4828d4687304041cf" candidate_uuid: "cfe7bca2991a42a5a845653ceda16955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b5bfd24bdb4f10a398777bb78e43ce"
I20260929 02:49:09.432574 6304 raft_consensus.cc:3037] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:09.432619 6446 raft_consensus.cc:3037] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:09.434479 6446 raft_consensus.cc:2454] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfe7bca2991a42a5a845653ceda16955 in term 1.
I20260929 02:49:09.434509 6304 raft_consensus.cc:2454] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfe7bca2991a42a5a845653ceda16955 in term 1.
I20260929 02:49:09.434867 6097 leader_election.cc:302] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b5bfd24bdb4f10a398777bb78e43ce, cfe7bca2991a42a5a845653ceda16955; no voters:
I20260929 02:49:09.435180 6524 raft_consensus.cc:2781] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:09.435333 6524 raft_consensus.cc:686] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [term 1 LEADER]: Becoming Leader. State: Replica: cfe7bca2991a42a5a845653ceda16955, State: Running, Role: LEADER
I20260929 02:49:09.435570 6524 consensus_queue.cc:237] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } }
I20260929 02:49:09.437111 6519 tablet_replica.cc:439] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: TxnStatusTablet state changed. Reason: New leader cfe7bca2991a42a5a845653ceda16955. Latest consensus state: current_term: 1 leader_uuid: "cfe7bca2991a42a5a845653ceda16955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } }
I20260929 02:49:09.437364 6519 tablet_replica.cc:442] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:49:09.437876 6556 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:49:09.438117 6556 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:49:09.438699 6009 catalog_manager.cc:5495] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfe7bca2991a42a5a845653ceda16955 (127.4.197.129). New cstate: current_term: 1 leader_uuid: "cfe7bca2991a42a5a845653ceda16955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } health_report { overall_health: UNKNOWN } } }
I20260929 02:49:09.500407 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.495253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.4.197.129:35855)
0929 02:49:09.495465 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.495471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.495497 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:49:09.495757 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.495764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.495835 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.495910 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:09.496273 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.497022 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.498281 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.498702 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.498805 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.499664 (+ 859us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:09.499782 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:09.499856 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.500002 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.500082 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:49:09.500388 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.495143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:35855 (local address 127.0.0.1:54654)
0929 02:49:09.495406 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:09.495429 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:09.495575 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:09.495951 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:09.495955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:09.495968 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:49:09.496141 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.496149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.497142 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.497147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.498148 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.498161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.498833 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.498838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.499442 (+ 604us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.499501 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:49:09.499878 (+ 377us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:49:09.499899 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 02:49:09.499966 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 02:49:09.500109 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 02:49:09.504521 6446 raft_consensus.cc:1261] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 FOLLOWER]: Refusing update from remote peer cfe7bca2991a42a5a845653ceda16955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:09.504777 6304 raft_consensus.cc:1261] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6 [term 1 FOLLOWER]: Refusing update from remote peer cfe7bca2991a42a5a845653ceda16955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:09.505091 6519 consensus_queue.cc:1035] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:09.505414 6524 consensus_queue.cc:1035] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:09.512526 6520 tablet_replica.cc:439] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: TxnStatusTablet state changed. Reason: New leader cfe7bca2991a42a5a845653ceda16955. Latest consensus state: current_term: 1 leader_uuid: "cfe7bca2991a42a5a845653ceda16955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } }
I20260929 02:49:09.512818 6520 tablet_replica.cc:442] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:09.513988 6521 tablet_replica.cc:439] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: TxnStatusTablet state changed. Reason: New leader cfe7bca2991a42a5a845653ceda16955. Latest consensus state: current_term: 1 leader_uuid: "cfe7bca2991a42a5a845653ceda16955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } }
I20260929 02:49:09.514248 6521 tablet_replica.cc:442] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:09.516436 6519 tablet_replica.cc:439] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cfe7bca2991a42a5a845653ceda16955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } }
I20260929 02:49:09.516788 6519 tablet_replica.cc:442] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:49:09.517333 6519 tablet_replica.cc:439] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cfe7bca2991a42a5a845653ceda16955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } }
I20260929 02:49:09.517592 6519 tablet_replica.cc:442] T a5e6c823c75643a4828d4687304041cf P cfe7bca2991a42a5a845653ceda16955: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:49:09.525224 6520 tablet_replica.cc:439] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cfe7bca2991a42a5a845653ceda16955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } }
I20260929 02:49:09.525492 6520 tablet_replica.cc:442] T a5e6c823c75643a4828d4687304041cf P 08b5bfd24bdb4f10a398777bb78e43ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:09.529412 6521 tablet_replica.cc:439] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cfe7bca2991a42a5a845653ceda16955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 } } peers { permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 } } peers { permanent_uuid: "08b5bfd24bdb4f10a398777bb78e43ce" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41947 } } }
I20260929 02:49:09.529650 6521 tablet_replica.cc:442] T a5e6c823c75643a4828d4687304041cf P 9c666adb2f1640239ae99408ad2f85d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:09.534749 6520 consensus_queue.cc:1035] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe7bca2991a42a5a845653ceda16955" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:09.538239 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.529594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.4.197.131:41947)
0929 02:49:09.530756 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.530763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.531307 (+ 544us) server_negotiation.cc:407] Connection header received
0929 02:49:09.531545 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.531550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.531650 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.531782 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:09.532279 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.532986 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.535552 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.536118 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.536233 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.536783 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:09.537559 (+ 776us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:09.537702 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.537819 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.537907 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":924,"threads_started":1}
I20260929 02:49:09.544045 6520 consensus_queue.cc:1035] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c666adb2f1640239ae99408ad2f85d6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:09.559891 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.552281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:51583 (local address 127.4.197.129:35855)
0929 02:49:09.552459 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.552465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.555422 (+ 2957us) server_negotiation.cc:407] Connection header received
0929 02:49:09.555591 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.555596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.555664 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.555745 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:09.556174 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.556919 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.558185 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.558656 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.558760 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.559195 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:09.559309 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:09.559387 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.559512 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.559587 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:49:09.559964 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.551840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:35855 (local address 127.4.197.131:51583)
0929 02:49:09.555318 (+ 3478us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:09.555334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:09.555469 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:09.555765 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:09.555770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:09.555793 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:49:09.556048 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.556056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.557030 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.557036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.558059 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.558070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.558820 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.558827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.558991 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.559043 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:49:09.559398 (+ 355us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:49:09.559409 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:49:09.559478 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 02:49:09.559601 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3345,"thread_start_us":3237,"threads_started":1}
I20260929 02:49:09.597090 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.590731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:36761 (local address 127.4.197.131:41947)
0929 02:49:09.590899 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.590905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.591107 (+ 202us) server_negotiation.cc:407] Connection header received
0929 02:49:09.591294 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.591299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.591374 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.591442 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:09.592024 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.593106 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.595234 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.595775 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.595891 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.596327 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:09.596461 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:09.596562 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.596696 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.596796 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:49:09.597194 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.590466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:41947 (local address 127.4.197.129:36761)
0929 02:49:09.591026 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:09.591048 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:09.591173 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:09.591471 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:09.591476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:09.591487 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:49:09.591850 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.591860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.593278 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.593285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.595083 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.595100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.595918 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.595925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.596082 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.596144 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:49:09.596560 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:49:09.596579 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 02:49:09.596660 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:49:09.596799 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":477,"threads_started":1}
I20260929 02:49:09.634699 6009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:09.635002 6009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:09.638625 6009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9697cda8f1a846d1a60c89457cbf9a27: Sending DeleteTablet for 3 replicas of tablet 7dcf80a981c0480db2557c89c2977ec3
I20260929 02:49:09.639780 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dcf80a981c0480db2557c89c2977ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:09 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:37480
I20260929 02:49:09.639787 6285 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dcf80a981c0480db2557c89c2977ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:09 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:33944
I20260929 02:49:09.640921 6146 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dcf80a981c0480db2557c89c2977ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:09 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:54642
I20260929 02:49:09.652204 6580 tablet_replica.cc:330] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: stopping tablet replica
I20260929 02:49:09.652455 6580 raft_consensus.cc:2229] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:09.653056 6580 raft_consensus.cc:2258] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:09.654304 6580 ts_tablet_manager.cc:1905] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:09.662464 6580 ts_tablet_manager.cc:1918] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:49:09.662644 6580 log.cc:1196] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/7dcf80a981c0480db2557c89c2977ec3
I20260929 02:49:09.663173 6580 ts_tablet_manager.cc:1939] T 7dcf80a981c0480db2557c89c2977ec3 P 08b5bfd24bdb4f10a398777bb78e43ce: Deleting consensus metadata
I20260929 02:49:09.664361 5994 catalog_manager.cc:4841] TS 08b5bfd24bdb4f10a398777bb78e43ce (127.4.197.131:41947): tablet 7dcf80a981c0480db2557c89c2977ec3 (table test-table [id=33e94b71009b48c3914b3063b825a8f8]) successfully deleted
I20260929 02:49:09.671792 6582 tablet_replica.cc:330] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: stopping tablet replica
I20260929 02:49:09.672045 6582 raft_consensus.cc:2229] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:09.673786 6581 tablet_replica.cc:330] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: stopping tablet replica
I20260929 02:49:09.674007 6581 raft_consensus.cc:2229] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:09.674242 6581 raft_consensus.cc:2258] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:09.675557 6582 raft_consensus.cc:2258] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:09.675902 6581 ts_tablet_manager.cc:1905] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:09.676640 6582 ts_tablet_manager.cc:1905] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:09.683259 6581 ts_tablet_manager.cc:1918] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:49:09.683501 6581 log.cc:1196] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/7dcf80a981c0480db2557c89c2977ec3
I20260929 02:49:09.683950 6581 ts_tablet_manager.cc:1939] T 7dcf80a981c0480db2557c89c2977ec3 P 9c666adb2f1640239ae99408ad2f85d6: Deleting consensus metadata
I20260929 02:49:09.684263 6582 ts_tablet_manager.cc:1918] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:49:09.684406 6582 log.cc:1196] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/7dcf80a981c0480db2557c89c2977ec3
I20260929 02:49:09.684830 6582 ts_tablet_manager.cc:1939] T 7dcf80a981c0480db2557c89c2977ec3 P cfe7bca2991a42a5a845653ceda16955: Deleting consensus metadata
I20260929 02:49:09.685483 5996 catalog_manager.cc:4841] TS 9c666adb2f1640239ae99408ad2f85d6 (127.4.197.130:41655): tablet 7dcf80a981c0480db2557c89c2977ec3 (table test-table [id=33e94b71009b48c3914b3063b825a8f8]) successfully deleted
I20260929 02:49:09.685791 5996 catalog_manager.cc:4841] TS cfe7bca2991a42a5a845653ceda16955 (127.4.197.129:35855): tablet 7dcf80a981c0480db2557c89c2977ec3 (table test-table [id=33e94b71009b48c3914b3063b825a8f8]) successfully deleted
Sep 29 02:49:09 dist-test-slave-t6c1 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:09.700140 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.674684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.4.197.129:35855)
0929 02:49:09.674988 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.674994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.675016 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:49:09.675083 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.675088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.675181 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.675285 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:09.687447 (+ 12162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.688213 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.689468 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.689892 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.689999 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.694367 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:09.694380 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:09.694398 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:09.694452 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:09.697616 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.698205 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.698213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.698215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.698341 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.698685 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.698690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.698693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.699044 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:09.699332 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.699575 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.699780 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 02:49:09.700851 6211 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54666
I20260929 02:49:09.740947 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.707830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.4.197.190:39073)
0929 02:49:09.708040 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.708044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.708588 (+ 544us) server_negotiation.cc:407] Connection header received
0929 02:49:09.708738 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.708744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.708814 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.708890 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:09.710398 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.711179 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.713221 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.713847 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.713960 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.716929 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:09.716936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:09.716940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:09.716979 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:09.720202 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.721038 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.721043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.721047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.721176 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.738803 (+ 17627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.738811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.738815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.739290 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:09.739604 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.740302 (+ 698us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.740580 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 02:49:09.742343 6009 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57688
I20260929 02:49:09.755647 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6080
I20260929 02:49:09.801441 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.759954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.4.197.129:35855)
0929 02:49:09.760163 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.760171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.777023 (+ 16852us) server_negotiation.cc:407] Connection header received
0929 02:49:09.777528 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.777534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.777618 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.777700 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:09.779516 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.780674 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.789887 (+ 9213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.790614 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.790735 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.794193 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:09.794202 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:09.794207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:09.794249 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:09.798240 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.799115 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.799121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.799125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.799274 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.799704 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.799709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.799712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.800048 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:09.800296 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.800837 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.801083 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 02:49:09.803073 6211 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54676
W20260929 02:49:09.803740 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6080: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:49:09.803938 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 6080
I20260929 02:49:09.833173 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6220
Sep 29 02:49:09 dist-test-slave-t6c1 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:49:09.849942 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.833895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.4.197.130:41655)
0929 02:49:09.834334 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.834339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.834355 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:49:09.834417 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.834421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.834482 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.834566 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:09.836275 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.836975 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.838813 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.839330 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.839434 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.844044 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:09.844059 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:09.844074 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:09.844115 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:09.846860 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.847722 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.847731 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.847735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.847935 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.848281 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.848286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.848289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.848832 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:09.849104 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.849321 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.849543 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":230,"threads_started":1}
W20260929 02:49:09.850623 6350 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33952
W20260929 02:49:09.851286 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6220: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:49:09.851450 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 6220
I20260929 02:49:09.865386 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6359
Sep 29 02:49:09 dist-test-slave-t6c1 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:49:09.880904 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.866105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.4.197.131:41947)
0929 02:49:09.866313 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.866319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.866339 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:49:09.866534 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.866540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.866620 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.866717 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:09.868052 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.868889 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.870094 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.870546 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.870659 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.874704 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:09.874717 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:09.874731 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:09.874771 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:09.877855 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.878522 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.878530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.878532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.878682 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.879315 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.879319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.879322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.879697 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:09.880028 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.880333 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.880578 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 02:49:09.881542 6489 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37500
W20260929 02:49:09.882165 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6359: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:49:09.882304 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 6359
I20260929 02:49:09.896456 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 5977
I20260929 02:49:09.897469 6049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:10.009277 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 5977
2026-09-29T02:49:10Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4452 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:10 dist-test-slave-t6c1 krb5kdc[6592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:10 dist-test-slave-t6c1 krb5kdc[6592](info): set up 2 sockets
Sep 29 02:49:10 dist-test-slave-t6c1 krb5kdc[6592](info): commencing operation
krb5kdc: starting...
W20260929 02:49:10.995064 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:49:11 dist-test-slave-t6c1 krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:49:11.160507 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36831
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:36831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:11.472513 6605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:11.472807 6605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:11.472922 6605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:11.480144 6605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:11.480245 6605 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:49:11.480306 6605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:11.480386 6605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:11.480461 6605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:11.480530 6605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:11.490300 6605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:36831
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:11.491591 6605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:11.492909 6605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:11.503798 6610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:11.503913 6611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:11.503912 6613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:11.504667 6605 server_base.cc:1034] running on GCE node
I20260929 02:49:11.505424 6605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:11.507082 6605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:11.508293 6605 hybrid_clock.cc:648] HybridClock initialized: now 1790650151508219 us; error 51 us; skew 500 ppm
I20260929 02:49:11.508620 6605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:11 dist-test-slave-t6c1 krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:11.511622 6605 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:49:11.513027 6605 webserver.cc:460] Webserver started at http://127.4.197.190:35485/ using document root <none> and password file <none>
I20260929 02:49:11.513731 6605 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:11.513818 6605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:11.514082 6605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:11.517175 6605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "419c732062fc49559fa22b96815f628a"
format_stamp: "Formatted at 2026-09-29 02:49:11 on dist-test-slave-t6c1"
server_key: "e3e1707c669ff0886b60717005d4e25f"
server_key_iv: "1f1e3c91b5239ded6cb35ace26a0299b"
server_key_version: "encryptionkey@0"
I20260929 02:49:11.517904 6605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "419c732062fc49559fa22b96815f628a"
format_stamp: "Formatted at 2026-09-29 02:49:11 on dist-test-slave-t6c1"
server_key: "e3e1707c669ff0886b60717005d4e25f"
server_key_iv: "1f1e3c91b5239ded6cb35ace26a0299b"
server_key_version: "encryptionkey@0"
I20260929 02:49:11.523308 6605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:49:11.526543 6620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:11.527913 6605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:11.528101 6605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "419c732062fc49559fa22b96815f628a"
format_stamp: "Formatted at 2026-09-29 02:49:11 on dist-test-slave-t6c1"
server_key: "e3e1707c669ff0886b60717005d4e25f"
server_key_iv: "1f1e3c91b5239ded6cb35ace26a0299b"
server_key_version: "encryptionkey@0"
I20260929 02:49:11.528327 6605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:11.542829 6605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:11.546710 6605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:11.546972 6605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:11.565088 6605 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:36831
I20260929 02:49:11.565107 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:36831 every 8 connection(s)
I20260929 02:49:11.566424 6605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:11.566994 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 6605
I20260929 02:49:11.567188 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:11.567790 4886 external_mini_cluster.cc:1428] Setting key c9cb5a564cb5daa2414a5b5a2ffec875
I20260929 02:49:11.571035 6682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:49:11 dist-test-slave-t6c1 krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:11.583981 6682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a: Bootstrap starting.
I20260929 02:49:11.588021 6682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:11.589133 6682 log.cc:824] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:11.591323 6685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:11.569708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.4.197.190:36831)
0929 02:49:11.570455 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:11.570468 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:11.570509 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:49:11.572157 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:11.572179 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:11.572571 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:11.572888 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:11.574597 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:11.576031 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:11.577753 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:11.578446 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:11.578572 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:11.583135 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:11.583146 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:11.583166 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:11.583215 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:11.586894 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:11.587585 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:11.587594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:11.587602 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:11.587825 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:11.588227 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:11.588233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:11.588236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:11.588792 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:11.589039 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:11.589510 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 02:49:11.590142 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":345,"threads_started":1}
I20260929 02:49:11.592269 6682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a: No bootstrap required, opened a new log
I20260929 02:49:11.599992 6682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419c732062fc49559fa22b96815f628a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36831 } }
I20260929 02:49:11.600383 6682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:11.600482 6682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419c732062fc49559fa22b96815f628a, State: Initialized, Role: FOLLOWER
I20260929 02:49:11.601126 6682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419c732062fc49559fa22b96815f628a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36831 } }
I20260929 02:49:11.601405 6682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:11.601536 6682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:11.601688 6682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:11.603926 6682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419c732062fc49559fa22b96815f628a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36831 } }
I20260929 02:49:11.604439 6682 leader_election.cc:302] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 419c732062fc49559fa22b96815f628a; no voters:
I20260929 02:49:11.605001 6682 leader_election.cc:288] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:11.605127 6687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:11.605870 6687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [term 1 LEADER]: Becoming Leader. State: Replica: 419c732062fc49559fa22b96815f628a, State: Running, Role: LEADER
I20260929 02:49:11.606397 6687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419c732062fc49559fa22b96815f628a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36831 } }
I20260929 02:49:11.607152 6682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:11.609441 6689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 419c732062fc49559fa22b96815f628a. Latest consensus state: current_term: 1 leader_uuid: "419c732062fc49559fa22b96815f628a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419c732062fc49559fa22b96815f628a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36831 } } }
I20260929 02:49:11.609920 6689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:11.610744 6696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:11.610575 6688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "419c732062fc49559fa22b96815f628a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419c732062fc49559fa22b96815f628a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36831 } } }
I20260929 02:49:11.610857 6688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:11.615870 6696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:11.624190 6696 catalog_manager.cc:1347] Generated new cluster ID: 9a6179b8a0cf473695095c9b6b18c501
I20260929 02:49:11.624312 6696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:11.644294 6696 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:11.646044 6696 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:11.657426 6696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a: Generated new TSK 0
I20260929 02:49:11.658277 6696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:49:11.704321 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36831
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:12.018268 6708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:12.018554 6708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:12.018695 6708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:12.025946 6708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:12.026078 6708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:12.026180 6708 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:49:12.026229 6708 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:49:12.026295 6708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:12.036392 6708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.197.190:36831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:12.037725 6708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:12.039134 6708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:12.053078 6713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:12.053180 6714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:12.053207 6716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:12.053704 6708 server_base.cc:1034] running on GCE node
I20260929 02:49:12.054352 6708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:12.055358 6708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:12.056543 6708 hybrid_clock.cc:648] HybridClock initialized: now 1790650152056486 us; error 56 us; skew 500 ppm
I20260929 02:49:12.056831 6708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:12 dist-test-slave-t6c1 krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650152, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:12.059728 6708 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:49:12.061295 6708 webserver.cc:460] Webserver started at http://127.4.197.129:37969/ using document root <none> and password file <none>
I20260929 02:49:12.061937 6708 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:12.062052 6708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:12.062335 6708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:12.065210 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "79e14a4d103f466e9ce863a58daeb3ac"
format_stamp: "Formatted at 2026-09-29 02:49:12 on dist-test-slave-t6c1"
server_key: "0fea849c97ca566ea03a8ee71acfe8d2"
server_key_iv: "d37e2883956b0014d37592f3304dc650"
server_key_version: "encryptionkey@0"
I20260929 02:49:12.065933 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "79e14a4d103f466e9ce863a58daeb3ac"
format_stamp: "Formatted at 2026-09-29 02:49:12 on dist-test-slave-t6c1"
server_key: "0fea849c97ca566ea03a8ee71acfe8d2"
server_key_iv: "d37e2883956b0014d37592f3304dc650"
server_key_version: "encryptionkey@0"
I20260929 02:49:12.070741 6708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 02:49:12.073953 6723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:12.075433 6708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:12.075596 6708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "79e14a4d103f466e9ce863a58daeb3ac"
format_stamp: "Formatted at 2026-09-29 02:49:12 on dist-test-slave-t6c1"
server_key: "0fea849c97ca566ea03a8ee71acfe8d2"
server_key_iv: "d37e2883956b0014d37592f3304dc650"
server_key_version: "encryptionkey@0"
I20260929 02:49:12.075798 6708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:12.094857 6708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:12.098912 6708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:12.099228 6708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:12.101828 6708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:12.101928 6708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:12.102042 6708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:12.102118 6708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:49:12 dist-test-slave-t6c1 krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650152, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:12.122447 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:12.102216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:44133 (local address 127.4.197.190:36831)
0929 02:49:12.102984 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:12.102990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:12.103926 (+ 936us) server_negotiation.cc:407] Connection header received
0929 02:49:12.105928 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:12.105934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:12.106010 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:12.106097 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:12.108124 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.109345 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.111681 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.112342 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.112475 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:12.116644 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:12.116652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:12.116657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:12.116700 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:12.119734 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:12.120352 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:12.120357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:12.120361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:12.120473 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:12.120859 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:12.120863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:12.120866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:12.121124 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:12.121248 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:12.121881 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 02:49:12.122104 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":498,"threads_started":1}
I20260929 02:49:12.123556 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:12.102904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36831 (local address 127.4.197.129:44133)
0929 02:49:12.103755 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:12.103799 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:12.105667 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:12.106374 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:12.106387 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:12.107132 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:12.107932 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:12.107960 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.109524 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.109529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:12.111544 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:12.111556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.112530 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.112535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:12.112734 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:12.113367 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:12.113400 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:12.116392 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:12.119854 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:12.119862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:12.119876 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:12.120237 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:12.120572 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:12.120577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:12.120581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:12.120753 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:12.121249 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:12.121256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:12.121562 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 02:49:12.121767 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 02:49:12.122225 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":339,"threads_started":1}
I20260929 02:49:12.143306 6708 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:33635
I20260929 02:49:12.143329 6843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:33635 every 8 connection(s)
I20260929 02:49:12.144763 6708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:12.146405 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 6708
I20260929 02:49:12.146600 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:12.147414 4886 external_mini_cluster.cc:1428] Setting key 25c0aeb6bde07c448a10a4cd30e5c2f8
I20260929 02:49:12.147087 6844 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36831
I20260929 02:49:12.148209 6844 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:12.148962 6844 heartbeater.cc:511] Master 127.4.197.190:36831 requested a full tablet report, sending...
I20260929 02:49:12.151422 6637 ts_manager.cc:194] Registered new tserver with Master: 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129:33635)
I20260929 02:49:12.152676 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:12.133039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:53687 (local address 127.4.197.190:36831)
0929 02:49:12.133194 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:12.133200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:12.139618 (+ 6418us) server_negotiation.cc:407] Connection header received
0929 02:49:12.139816 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:12.139822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:12.139905 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:12.140011 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:12.141399 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.142249 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.144270 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.144692 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.144783 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:12.147045 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:12.147051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:12.147056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:12.147096 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:12.150185 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:12.150794 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:12.150800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:12.150804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:12.150959 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:12.151278 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:12.151282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:12.151285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:12.151568 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:12.151705 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:12.152130 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 02:49:12.152337 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:49:12.152719 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:12.132558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36831 (local address 127.4.197.129:53687)
0929 02:49:12.139488 (+ 6930us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:12.139511 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:12.139687 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:12.140066 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:12.140071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:12.140750 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:12.141216 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:12.141235 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.142427 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.142433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:12.144142 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:12.144155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.144892 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.144899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:12.145076 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:12.145768 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:12.145805 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:12.146847 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:12.150326 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:12.150329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:12.150332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:12.150671 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:12.151057 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:12.151061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:12.151063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:12.151176 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:12.151752 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:12.151755 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:12.151901 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 02:49:12.152011 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 02:49:12.152296 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6751,"thread_start_us":6565,"threads_started":1}
I20260929 02:49:12.153312 6637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:44133
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:49:12.183774 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36831
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:12.497510 6848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:12.497798 6848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:12.497941 6848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:12.505002 6848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:12.505108 6848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:12.505224 6848 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:49:12.505295 6848 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:49:12.505394 6848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:12.514791 6848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.197.190:36831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:12.516021 6848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:12.517248 6848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:12.529390 6853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:12.529423 6854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:12.529459 6856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:12.530712 6848 server_base.cc:1034] running on GCE node
I20260929 02:49:12.531417 6848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:12.532475 6848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:12.533668 6848 hybrid_clock.cc:648] HybridClock initialized: now 1790650152533608 us; error 58 us; skew 500 ppm
I20260929 02:49:12.533975 6848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:12 dist-test-slave-t6c1 krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650152, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:12.537087 6848 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:49:12.538600 6848 webserver.cc:460] Webserver started at http://127.4.197.130:43427/ using document root <none> and password file <none>
I20260929 02:49:12.539336 6848 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:12.539450 6848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:12.539742 6848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:12.542758 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "4ccec73cc5dc4c9581797eea02be54c4"
format_stamp: "Formatted at 2026-09-29 02:49:12 on dist-test-slave-t6c1"
server_key: "425330534570e31f69e7d587a0550ab4"
server_key_iv: "441c3d1a6f8d21fab47a6118f0351841"
server_key_version: "encryptionkey@0"
I20260929 02:49:12.543610 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "4ccec73cc5dc4c9581797eea02be54c4"
format_stamp: "Formatted at 2026-09-29 02:49:12 on dist-test-slave-t6c1"
server_key: "425330534570e31f69e7d587a0550ab4"
server_key_iv: "441c3d1a6f8d21fab47a6118f0351841"
server_key_version: "encryptionkey@0"
I20260929 02:49:12.548764 6848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:49:12.552170 6863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:12.553462 6848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:12.553606 6848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "4ccec73cc5dc4c9581797eea02be54c4"
format_stamp: "Formatted at 2026-09-29 02:49:12 on dist-test-slave-t6c1"
server_key: "425330534570e31f69e7d587a0550ab4"
server_key_iv: "441c3d1a6f8d21fab47a6118f0351841"
server_key_version: "encryptionkey@0"
I20260929 02:49:12.553800 6848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:12.587616 6848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:12.591395 6848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:12.591648 6848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:12.594221 6848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:12.594326 6848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:12.594439 6848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:12.594511 6848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:49:12 dist-test-slave-t6c1 krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650152, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:12.617910 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:12.594859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:40919 (local address 127.4.197.190:36831)
0929 02:49:12.595076 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:12.595081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:12.596489 (+ 1408us) server_negotiation.cc:407] Connection header received
0929 02:49:12.599398 (+ 2909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:12.599404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:12.599493 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:12.599592 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:12.601782 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.602730 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.605260 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.605953 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.606108 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:12.611261 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:12.611267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:12.611271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:12.611315 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:12.614438 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:12.615205 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:12.615211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:12.615215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:12.615375 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:12.615971 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:12.615976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:12.615978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:12.616285 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:12.616403 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:12.617474 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0929 02:49:12.617654 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 02:49:12.619879 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:12.595430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36831 (local address 127.4.197.130:40919)
0929 02:49:12.596292 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:12.596361 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:12.598994 (+ 2633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:12.599960 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:12.599972 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:12.600696 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:12.601607 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:12.601631 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.602982 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.602988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:12.605054 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:12.605077 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.606240 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.606252 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:12.606491 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:12.607276 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:12.607312 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:12.610994 (+ 3682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:12.614577 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:12.614584 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:12.614597 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:12.615076 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:12.615509 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:12.615515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:12.615519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:12.615834 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:12.616406 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:12.616415 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:12.617007 (+ 592us) client_negotiation.cc:769] Sending connection context
0929 02:49:12.617357 (+ 350us) client_negotiation.cc:240] Negotiation successful
0929 02:49:12.618000 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":326,"threads_started":1}
I20260929 02:49:12.634882 6848 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:42523
I20260929 02:49:12.635032 6982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:42523 every 8 connection(s)
I20260929 02:49:12.636435 6848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:12.638645 6983 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36831
I20260929 02:49:12.638852 6983 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:12.639659 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 6848
I20260929 02:49:12.639675 6983 heartbeater.cc:511] Master 127.4.197.190:36831 requested a full tablet report, sending...
I20260929 02:49:12.639817 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:12.640286 4886 external_mini_cluster.cc:1428] Setting key 68791a796f5ac93543cdffad8a7f209e
I20260929 02:49:12.641206 6636 ts_manager.cc:194] Registered new tserver with Master: 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523)
I20260929 02:49:12.642009 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:12.627246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36831 (local address 127.4.197.130:37113)
0929 02:49:12.627968 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:12.627988 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:12.628231 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:12.628632 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:12.628641 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:12.629396 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:12.629809 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:12.629825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.631248 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.631257 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:12.632943 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:12.632961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.633728 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.633734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:12.633902 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:12.634562 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:12.634586 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:12.635556 (+ 970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:12.639146 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:12.639151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:12.639155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:12.639703 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:12.640148 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:12.640152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:12.640155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:12.640304 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:12.640923 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:12.640928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:12.641108 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 02:49:12.641271 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 02:49:12.641576 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":452,"threads_started":1}
I20260929 02:49:12.642438 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:12.627152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:37113 (local address 127.4.197.190:36831)
0929 02:49:12.627341 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:12.627347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:12.628088 (+ 741us) server_negotiation.cc:407] Connection header received
0929 02:49:12.628391 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:12.628397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:12.628478 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:12.628578 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:12.630014 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.631056 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.633082 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:12.633537 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:12.633649 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:12.635733 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:12.635739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:12.635745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:12.635797 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:12.639032 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:12.639853 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:12.639859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:12.639863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:12.640021 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:12.640456 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:12.640461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:12.640465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:12.640780 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:12.640955 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:12.641880 (+ 925us) server_negotiation.cc:299] Negotiation successful
0929 02:49:12.642117 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:49:12.642416 6636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:40919
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:49:12.679746 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36831
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:12.992228 6987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:12.992523 6987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:12.992647 6987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:12.999651 6987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:12.999764 6987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:12.999887 6987 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:49:12.999955 6987 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:49:13.000054 6987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:13.009467 6987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.197.190:36831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:13.010664 6987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:13.011961 6987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:13.024075 6992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:13.024104 6995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:13.024024 6993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:13.024909 6987 server_base.cc:1034] running on GCE node
I20260929 02:49:13.025490 6987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:13.026394 6987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:13.027591 6987 hybrid_clock.cc:648] HybridClock initialized: now 1790650153027545 us; error 40 us; skew 500 ppm
I20260929 02:49:13.027894 6987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:13 dist-test-slave-t6c1 krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:13.030999 6987 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:49:13.032744 6987 webserver.cc:460] Webserver started at http://127.4.197.131:32891/ using document root <none> and password file <none>
I20260929 02:49:13.033416 6987 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:13.033531 6987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:13.033812 6987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:13.036707 6987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "f694ac5962cb487dbd62d8c42efafe8a"
format_stamp: "Formatted at 2026-09-29 02:49:13 on dist-test-slave-t6c1"
server_key: "af514c8fdd321d0247c61a143a99c9ad"
server_key_iv: "bb844eb7030a464f32a8b77f7623e418"
server_key_version: "encryptionkey@0"
I20260929 02:49:13.037420 6987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "f694ac5962cb487dbd62d8c42efafe8a"
format_stamp: "Formatted at 2026-09-29 02:49:13 on dist-test-slave-t6c1"
server_key: "af514c8fdd321d0247c61a143a99c9ad"
server_key_iv: "bb844eb7030a464f32a8b77f7623e418"
server_key_version: "encryptionkey@0"
I20260929 02:49:13.042358 6987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:49:13.045855 7002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:13.047241 6987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:49:13.047387 6987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "f694ac5962cb487dbd62d8c42efafe8a"
format_stamp: "Formatted at 2026-09-29 02:49:13 on dist-test-slave-t6c1"
server_key: "af514c8fdd321d0247c61a143a99c9ad"
server_key_iv: "bb844eb7030a464f32a8b77f7623e418"
server_key_version: "encryptionkey@0"
I20260929 02:49:13.047554 6987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:13.068454 6987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:13.072727 6987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:13.072999 6987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:13.075670 6987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:13.075770 6987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:13.075881 6987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:13.075953 6987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:49:13 dist-test-slave-t6c1 krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:13.098129 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.076177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:44957 (local address 127.4.197.190:36831)
0929 02:49:13.076416 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.076423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.077856 (+ 1433us) server_negotiation.cc:407] Connection header received
0929 02:49:13.080112 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.080119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.080206 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.080278 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:13.082642 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.083627 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.085926 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.086467 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.086599 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.091485 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:13.091492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:13.091497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:13.091543 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:13.094421 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:13.095461 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:13.095467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:13.095471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:13.095637 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:13.096189 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:13.096194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:13.096198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:13.096611 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:13.096748 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.097552 (+ 804us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.097782 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:49:13.099256 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.076860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36831 (local address 127.4.197.131:44957)
0929 02:49:13.077681 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.077725 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.079750 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.080825 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.080835 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.081584 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:13.082450 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.082474 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.083769 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.083777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.085789 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.085805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.086887 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.086892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.087174 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.087921 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:13.087962 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:13.091225 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:13.094564 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:13.094575 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:13.094600 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:13.095329 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:13.095753 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:13.095758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:13.095762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:13.096079 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:13.096741 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:13.096752 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:13.097159 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.097413 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.097943 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":331,"threads_started":1}
I20260929 02:49:13.115549 6987 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:41139
I20260929 02:49:13.115566 7121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:41139 every 8 connection(s)
I20260929 02:49:13.116966 6987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:13.118722 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.105217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36831 (local address 127.4.197.131:59029)
0929 02:49:13.105716 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.105731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.105859 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.106284 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.106289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.106894 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:13.107224 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.107233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.108716 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.108722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.110231 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.110246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.111150 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.111157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.111331 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.111951 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:13.111974 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:13.113009 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:13.116236 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:13.116240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:13.116242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:13.116614 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:13.116970 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:13.116973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:13.116975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:13.117118 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:13.117687 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:13.117691 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:13.117882 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.118042 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.118324 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":268,"threads_started":1}
I20260929 02:49:13.118741 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.105116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:59029 (local address 127.4.197.190:36831)
0929 02:49:13.105248 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.105252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.105836 (+ 584us) server_negotiation.cc:407] Connection header received
0929 02:49:13.105976 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.105981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.106049 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.106129 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:13.107356 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.108544 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.110457 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.110968 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.111086 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.113251 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:13.113257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:13.113271 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:13.113300 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:13.116125 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:13.116723 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:13.116728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:13.116731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:13.116878 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:13.117229 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:13.117234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:13.117238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:13.117573 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:13.117729 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.118149 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.118370 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 02:49:13.118911 7122 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36831
I20260929 02:49:13.119151 7122 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:13.120468 7122 heartbeater.cc:511] Master 127.4.197.190:36831 requested a full tablet report, sending...
I20260929 02:49:13.121686 6636 ts_manager.cc:194] Registered new tserver with Master: f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139)
I20260929 02:49:13.122650 6636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:44957
I20260929 02:49:13.125715 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 6987
I20260929 02:49:13.125892 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:13.126336 4886 external_mini_cluster.cc:1428] Setting key 857b66a5f71837286dec303e10b3e387
I20260929 02:49:13.129176 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:49:13 dist-test-slave-t6c1 krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260929 02:49:13.156803 6844 heartbeater.cc:503] Master 127.4.197.190:36831 was elected leader, sending a full tablet report...
Sep 29 02:49:13 dist-test-slave-t6c1 krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:13.173985 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.161199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.4.197.190:36831)
0929 02:49:13.161417 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.161423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.161640 (+ 217us) server_negotiation.cc:407] Connection header received
0929 02:49:13.161829 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.161833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.161914 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.162000 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:13.163350 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.164065 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.165203 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.165608 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.165696 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.169347 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:13.169352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:13.169355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:13.169392 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:13.171672 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:13.172271 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:13.172274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:13.172276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:13.172402 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:13.172717 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:13.172721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:13.172724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:13.173017 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:13.173204 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.173477 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.173660 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:49:13.177860 6636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:13.184921 6636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:13.223908 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.197440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.4.197.129:33635)
0929 02:49:13.214019 (+ 16579us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.214029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.215299 (+ 1270us) server_negotiation.cc:407] Connection header received
0929 02:49:13.215422 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.215431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.215953 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.216156 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:13.217528 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.218669 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.221219 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.222582 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.222753 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.223009 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.223448 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.223614 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16425,"thread_start_us":16286,"threads_started":1}
I20260929 02:49:13.224642 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.196853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.4.197.131:41139)
0929 02:49:13.215763 (+ 18910us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.215771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.215802 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:49:13.215908 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.215916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.216155 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.216342 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:13.217269 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.218962 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.221677 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.222710 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.222823 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.223073 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.224243 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.224374 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18755,"thread_start_us":18588,"threads_started":1}
I20260929 02:49:13.224730 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.197892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.4.197.130:42523)
0929 02:49:13.214450 (+ 16558us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.214461 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.215562 (+ 1101us) server_negotiation.cc:407] Connection header received
0929 02:49:13.215682 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.215689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.215954 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.216157 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:13.217459 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.218952 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.221645 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.223009 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.223146 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.223359 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.224152 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.224327 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16398,"thread_start_us":16253,"threads_started":1}
I20260929 02:49:13.225081 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.197003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:42523 (local address 127.0.0.1:52012)
0929 02:49:13.215300 (+ 18297us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.215317 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.215437 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.216405 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.216413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.216454 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:13.217277 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.217295 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.219109 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.219114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.221491 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.221505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.223258 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.223266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.223473 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.223508 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.223602 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.223729 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18184,"mutex_wait_us":152,"spinlock_wait_cycles":1280,"thread_start_us":18051,"threads_started":1}
I20260929 02:49:13.225508 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.196690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:33635 (local address 127.0.0.1:39038)
0929 02:49:13.214957 (+ 18267us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.214981 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.215124 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.216405 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.216413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.216460 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:13.217346 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.217357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.218836 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.218842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.221060 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.221076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.222783 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.222789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.223066 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.223117 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.223327 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.224257 (+ 930us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18113,"mutex_wait_us":186,"spinlock_wait_cycles":2304,"thread_start_us":17961,"threads_started":1}
I20260929 02:49:13.224866 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.196593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:41139 (local address 127.0.0.1:52200)
0929 02:49:13.211463 (+ 14870us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.211528 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.211748 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.216628 (+ 4880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.216633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.216676 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:13.217045 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.217058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.219288 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.219293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.221499 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.221507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.223231 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.223238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.223417 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.223447 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.223519 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.223679 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14526,"mutex_wait_us":52,"thread_start_us":15359,"threads_started":1}
I20260929 02:49:13.228833 6774 tablet_service.cc:1467] Processing CreateTablet for tablet c901ad0a0c8841edbad9facd78ce4e35 (DEFAULT_TABLE table=test-table [id=43200604f9e946138a061e00917c7bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:13.228963 6913 tablet_service.cc:1467] Processing CreateTablet for tablet c901ad0a0c8841edbad9facd78ce4e35 (DEFAULT_TABLE table=test-table [id=43200604f9e946138a061e00917c7bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:13.228963 7052 tablet_service.cc:1467] Processing CreateTablet for tablet c901ad0a0c8841edbad9facd78ce4e35 (DEFAULT_TABLE table=test-table [id=43200604f9e946138a061e00917c7bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:13.230077 6774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c901ad0a0c8841edbad9facd78ce4e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:13.230280 7052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c901ad0a0c8841edbad9facd78ce4e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:13.230293 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c901ad0a0c8841edbad9facd78ce4e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:13.243543 7141 tablet_bootstrap.cc:492] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: Bootstrap starting.
I20260929 02:49:13.245281 7143 tablet_bootstrap.cc:492] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: Bootstrap starting.
I20260929 02:49:13.247664 7142 tablet_bootstrap.cc:492] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: Bootstrap starting.
I20260929 02:49:13.248008 7141 tablet_bootstrap.cc:654] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:13.248603 7143 tablet_bootstrap.cc:654] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:13.249382 7141 log.cc:824] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:13.249752 7143 log.cc:824] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:13.251256 7142 tablet_bootstrap.cc:654] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:13.252384 7142 log.cc:824] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:13.252547 7141 tablet_bootstrap.cc:492] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: No bootstrap required, opened a new log
I20260929 02:49:13.252856 7141 ts_tablet_manager.cc:1397] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260929 02:49:13.255012 7143 tablet_bootstrap.cc:492] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: No bootstrap required, opened a new log
I20260929 02:49:13.255275 7143 ts_tablet_manager.cc:1397] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260929 02:49:13.255769 7142 tablet_bootstrap.cc:492] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: No bootstrap required, opened a new log
I20260929 02:49:13.255999 7142 ts_tablet_manager.cc:1397] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:49:13.263170 7141 raft_consensus.cc:348] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } }
I20260929 02:49:13.263677 7141 raft_consensus.cc:374] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:13.263777 7141 raft_consensus.cc:729] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e14a4d103f466e9ce863a58daeb3ac, State: Initialized, Role: FOLLOWER
I20260929 02:49:13.263582 7142 raft_consensus.cc:348] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } }
I20260929 02:49:13.264056 7142 raft_consensus.cc:374] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:13.264153 7142 raft_consensus.cc:729] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f694ac5962cb487dbd62d8c42efafe8a, State: Initialized, Role: FOLLOWER
I20260929 02:49:13.264567 7141 consensus_queue.cc:260] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } }
I20260929 02:49:13.264775 7142 consensus_queue.cc:260] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } }
I20260929 02:49:13.265344 7143 raft_consensus.cc:348] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } }
I20260929 02:49:13.265790 7143 raft_consensus.cc:374] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:13.265878 7143 raft_consensus.cc:729] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ccec73cc5dc4c9581797eea02be54c4, State: Initialized, Role: FOLLOWER
I20260929 02:49:13.266496 7143 consensus_queue.cc:260] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } }
I20260929 02:49:13.267349 7122 heartbeater.cc:503] Master 127.4.197.190:36831 was elected leader, sending a full tablet report...
I20260929 02:49:13.267872 7142 ts_tablet_manager.cc:1428] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 02:49:13.269596 7141 ts_tablet_manager.cc:1428] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260929 02:49:13.270756 6983 heartbeater.cc:503] Master 127.4.197.190:36831 was elected leader, sending a full tablet report...
I20260929 02:49:13.271373 7143 ts_tablet_manager.cc:1428] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260929 02:49:13.369920 7123 tablet.cc:2369] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:13.377425 7148 raft_consensus.cc:482] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:13.377731 7148 raft_consensus.cc:504] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } }
I20260929 02:49:13.379444 7148 leader_election.cc:288] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129:33635), 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523)
I20260929 02:49:13.384954 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.379915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:33635 (local address 127.4.197.131:34377)
0929 02:49:13.380246 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.380262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.380410 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.380766 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.380771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.380799 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:13.381013 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.381025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.381957 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.381963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.383281 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.383292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.384287 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.384293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.384463 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.384486 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.384564 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.384688 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260929 02:49:13.384944 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.379862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:34377 (local address 127.4.197.129:33635)
0929 02:49:13.380078 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.380083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.380367 (+ 284us) server_negotiation.cc:407] Connection header received
0929 02:49:13.380570 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.380576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.380656 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.380733 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:13.381131 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.381825 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.383420 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.384149 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.384287 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.384358 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.384598 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.384721 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 02:49:13.385842 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c901ad0a0c8841edbad9facd78ce4e35" candidate_uuid: "f694ac5962cb487dbd62d8c42efafe8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e14a4d103f466e9ce863a58daeb3ac" is_pre_election: true
I20260929 02:49:13.386010 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.379971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:35849 (local address 127.4.197.130:42523)
0929 02:49:13.380176 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.380182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.380823 (+ 641us) server_negotiation.cc:407] Connection header received
0929 02:49:13.381064 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.381069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.381134 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.381218 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:13.381750 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.382477 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.384253 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.385032 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.385216 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.385304 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.385645 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.385772 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:49:13.386082 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.379956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:42523 (local address 127.4.197.131:35849)
0929 02:49:13.380697 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.380713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.380912 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.381243 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.381248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.381272 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:13.381577 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.381587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.382639 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.382645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.384076 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.384089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.385336 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.385341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.385497 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.385520 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.385599 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.385711 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"spinlock_wait_cycles":36096,"thread_start_us":429,"threads_started":1}
I20260929 02:49:13.386399 6798 raft_consensus.cc:2454] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f694ac5962cb487dbd62d8c42efafe8a in term 0.
I20260929 02:49:13.387085 7005 leader_election.cc:302] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79e14a4d103f466e9ce863a58daeb3ac, f694ac5962cb487dbd62d8c42efafe8a; no voters:
I20260929 02:49:13.387147 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c901ad0a0c8841edbad9facd78ce4e35" candidate_uuid: "f694ac5962cb487dbd62d8c42efafe8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ccec73cc5dc4c9581797eea02be54c4" is_pre_election: true
I20260929 02:49:13.387449 7148 raft_consensus.cc:2781] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:13.387573 7148 raft_consensus.cc:482] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:13.387669 7148 raft_consensus.cc:3037] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:13.387609 6933 raft_consensus.cc:2454] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f694ac5962cb487dbd62d8c42efafe8a in term 0.
W20260929 02:49:13.388813 6984 tablet.cc:2369] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:13.390120 7148 raft_consensus.cc:504] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } }
I20260929 02:49:13.390726 7148 leader_election.cc:288] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [CANDIDATE]: Term 1 election: Requested vote from peers 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129:33635), 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523)
I20260929 02:49:13.391129 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c901ad0a0c8841edbad9facd78ce4e35" candidate_uuid: "f694ac5962cb487dbd62d8c42efafe8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e14a4d103f466e9ce863a58daeb3ac"
I20260929 02:49:13.391191 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c901ad0a0c8841edbad9facd78ce4e35" candidate_uuid: "f694ac5962cb487dbd62d8c42efafe8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ccec73cc5dc4c9581797eea02be54c4"
I20260929 02:49:13.391322 6798 raft_consensus.cc:3037] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:13.391341 6933 raft_consensus.cc:3037] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:13.393347 6933 raft_consensus.cc:2454] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f694ac5962cb487dbd62d8c42efafe8a in term 1.
I20260929 02:49:13.393467 6798 raft_consensus.cc:2454] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f694ac5962cb487dbd62d8c42efafe8a in term 1.
I20260929 02:49:13.393723 7006 leader_election.cc:302] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ccec73cc5dc4c9581797eea02be54c4, f694ac5962cb487dbd62d8c42efafe8a; no voters:
I20260929 02:49:13.394016 7148 raft_consensus.cc:2781] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:13.394402 7148 raft_consensus.cc:686] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 1 LEADER]: Becoming Leader. State: Replica: f694ac5962cb487dbd62d8c42efafe8a, State: Running, Role: LEADER
I20260929 02:49:13.394800 7148 consensus_queue.cc:237] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } }
W20260929 02:49:13.397300 6845 tablet.cc:2369] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:13.398872 6636 catalog_manager.cc:5495] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a reported cstate change: term changed from 0 to 1, leader changed from <none> to f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131). New cstate: current_term: 1 leader_uuid: "f694ac5962cb487dbd62d8c42efafe8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:49:13 dist-test-slave-t6c1 krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:13.445171 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.433236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.4.197.190:36831)
0929 02:49:13.433423 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.433427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.434028 (+ 601us) server_negotiation.cc:407] Connection header received
0929 02:49:13.434162 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.434166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.434226 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.434303 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:13.435453 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.436150 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.437350 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.437771 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.437860 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.440783 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:13.440788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:13.440791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:13.440818 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:13.443093 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:13.443659 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:13.443663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:13.443665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:13.443771 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:13.444061 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:13.444064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:13.444066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:13.444303 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:13.444408 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.444744 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.444938 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":63}
I20260929 02:49:13.445197 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.433404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36831 (local address 127.0.0.1:45120)
0929 02:49:13.433925 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.433945 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.434076 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.434334 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.434338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.435069 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:13.435329 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.435338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.436262 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.436266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.437229 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.437239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.437931 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.437936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.438053 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.438605 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:13.438622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:13.440648 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:13.443203 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:13.443212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:13.443217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:13.443573 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:13.443843 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:13.443845 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:13.443848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:13.443983 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:13.444396 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:13.444401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:13.444537 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.444661 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.444870 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260929 02:49:13.449709 6636 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:45120:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:49:13.450690 6636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:13.457160 6774 tablet_service.cc:1467] Processing CreateTablet for tablet 44a40ff21e8f4331b9ae41f39372bc1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c19de2b6df424a28a5d36f9275154553]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:49:13.457370 7052 tablet_service.cc:1467] Processing CreateTablet for tablet 44a40ff21e8f4331b9ae41f39372bc1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c19de2b6df424a28a5d36f9275154553]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:49:13.457370 6913 tablet_service.cc:1467] Processing CreateTablet for tablet 44a40ff21e8f4331b9ae41f39372bc1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c19de2b6df424a28a5d36f9275154553]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:49:13.458170 6774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a40ff21e8f4331b9ae41f39372bc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:13.458225 7052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a40ff21e8f4331b9ae41f39372bc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:13.458249 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a40ff21e8f4331b9ae41f39372bc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:13.463613 7141 tablet_bootstrap.cc:492] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: Bootstrap starting.
I20260929 02:49:13.465065 7141 tablet_bootstrap.cc:654] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:13.465181 7143 tablet_bootstrap.cc:492] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: Bootstrap starting.
I20260929 02:49:13.465181 7142 tablet_bootstrap.cc:492] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: Bootstrap starting.
I20260929 02:49:13.466776 7142 tablet_bootstrap.cc:654] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:13.466776 7143 tablet_bootstrap.cc:654] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:13.469887 7141 tablet_bootstrap.cc:492] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: No bootstrap required, opened a new log
I20260929 02:49:13.469965 7143 tablet_bootstrap.cc:492] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: No bootstrap required, opened a new log
I20260929 02:49:13.470045 7141 ts_tablet_manager.cc:1397] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 02:49:13.470088 7143 ts_tablet_manager.cc:1397] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260929 02:49:13.470148 7142 tablet_bootstrap.cc:492] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: No bootstrap required, opened a new log
I20260929 02:49:13.470284 7142 ts_tablet_manager.cc:1397] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260929 02:49:13.470783 7141 raft_consensus.cc:348] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.470784 7143 raft_consensus.cc:348] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.471047 7141 raft_consensus.cc:374] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:13.471067 7143 raft_consensus.cc:374] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:13.470891 7142 raft_consensus.cc:348] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.471144 7141 raft_consensus.cc:729] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e14a4d103f466e9ce863a58daeb3ac, State: Initialized, Role: FOLLOWER
I20260929 02:49:13.471146 7142 raft_consensus.cc:374] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:13.471148 7143 raft_consensus.cc:729] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ccec73cc5dc4c9581797eea02be54c4, State: Initialized, Role: FOLLOWER
I20260929 02:49:13.471237 7142 raft_consensus.cc:729] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f694ac5962cb487dbd62d8c42efafe8a, State: Initialized, Role: FOLLOWER
I20260929 02:49:13.471359 7143 consensus_queue.cc:260] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.471427 7141 consensus_queue.cc:260] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.471448 7142 consensus_queue.cc:260] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.471966 7143 ts_tablet_manager.cc:1428] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:13.471969 7142 ts_tablet_manager.cc:1428] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:49:13.471819 7149 tablet_replica.cc:439] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } }
I20260929 02:49:13.472185 7149 tablet_replica.cc:442] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:13.472396 7148 tablet_replica.cc:439] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } }
I20260929 02:49:13.472663 7148 tablet_replica.cc:442] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:13.472999 7141 ts_tablet_manager.cc:1428] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:49:13.473023 7147 tablet_replica.cc:439] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } }
I20260929 02:49:13.473803 7147 tablet_replica.cc:442] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:13.576105 7147 raft_consensus.cc:482] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:13.576366 7147 raft_consensus.cc:504] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.577970 7147 leader_election.cc:288] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523), f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139)
I20260929 02:49:13.584681 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.578433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:59563 (local address 127.4.197.131:41139)
0929 02:49:13.578630 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.578638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.579411 (+ 773us) server_negotiation.cc:407] Connection header received
0929 02:49:13.579583 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.579587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.579648 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.579721 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:13.580342 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.581055 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.582989 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.583762 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.583927 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.584006 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.584296 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.584435 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:49:13.584798 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.578469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:41139 (local address 127.4.197.129:59563)
0929 02:49:13.579279 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.579298 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.579452 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.579746 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.579750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.579781 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:13.580167 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.580182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.581183 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.581189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.582793 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.582806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.583971 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.583977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.584140 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.584166 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.584257 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.584405 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":80,"thread_start_us":532,"threads_started":1}
I20260929 02:49:13.585095 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.578510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:34229 (local address 127.4.197.130:42523)
0929 02:49:13.578688 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.578695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.579571 (+ 876us) server_negotiation.cc:407] Connection header received
0929 02:49:13.579761 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.579765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.579829 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.579926 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:13.580536 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.581313 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.583212 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.584069 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.584251 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.584336 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.584652 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.584790 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:49:13.585268 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.578489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:42523 (local address 127.4.197.129:34229)
0929 02:49:13.579449 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.579465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.579611 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.579962 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.579966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.579991 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:13.580374 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.580387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.581465 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.581471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.583042 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.583056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.584309 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.584316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.584484 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.584510 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.584605 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.584765 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":3,"spinlock_wait_cycles":40192,"thread_start_us":454,"threads_started":1}
I20260929 02:49:13.585947 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a40ff21e8f4331b9ae41f39372bc1f" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ccec73cc5dc4c9581797eea02be54c4" is_pre_election: true
I20260929 02:49:13.586041 7072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a40ff21e8f4331b9ae41f39372bc1f" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f694ac5962cb487dbd62d8c42efafe8a" is_pre_election: true
I20260929 02:49:13.586228 6933 raft_consensus.cc:2454] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 0.
I20260929 02:49:13.586391 7072 raft_consensus.cc:2454] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 0.
I20260929 02:49:13.586750 6727 leader_election.cc:302] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ccec73cc5dc4c9581797eea02be54c4, 79e14a4d103f466e9ce863a58daeb3ac; no voters:
I20260929 02:49:13.587215 7147 raft_consensus.cc:2781] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:13.587327 7147 raft_consensus.cc:482] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:13.587417 7147 raft_consensus.cc:3037] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:13.589553 7147 raft_consensus.cc:504] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.590122 7147 leader_election.cc:288] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 election: Requested vote from peers 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523), f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139)
I20260929 02:49:13.590708 7072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a40ff21e8f4331b9ae41f39372bc1f" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f694ac5962cb487dbd62d8c42efafe8a"
I20260929 02:49:13.590690 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a40ff21e8f4331b9ae41f39372bc1f" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ccec73cc5dc4c9581797eea02be54c4"
I20260929 02:49:13.590909 7072 raft_consensus.cc:3037] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:13.590965 6933 raft_consensus.cc:3037] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:13.593825 7072 raft_consensus.cc:2454] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 1.
I20260929 02:49:13.593809 6933 raft_consensus.cc:2454] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 1.
I20260929 02:49:13.594285 6727 leader_election.cc:302] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ccec73cc5dc4c9581797eea02be54c4, 79e14a4d103f466e9ce863a58daeb3ac, f694ac5962cb487dbd62d8c42efafe8a; no voters:
I20260929 02:49:13.594630 7147 raft_consensus.cc:2781] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:13.595105 7147 raft_consensus.cc:686] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [term 1 LEADER]: Becoming Leader. State: Replica: 79e14a4d103f466e9ce863a58daeb3ac, State: Running, Role: LEADER
I20260929 02:49:13.595551 7147 consensus_queue.cc:237] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.598125 7164 tablet_replica.cc:439] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: TxnStatusTablet state changed. Reason: New leader 79e14a4d103f466e9ce863a58daeb3ac. Latest consensus state: current_term: 1 leader_uuid: "79e14a4d103f466e9ce863a58daeb3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } }
I20260929 02:49:13.598448 7164 tablet_replica.cc:442] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:49:13.599159 7166 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:49:13.599403 7166 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:49:13.600116 6635 catalog_manager.cc:5495] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129). New cstate: current_term: 1 leader_uuid: "79e14a4d103f466e9ce863a58daeb3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } health_report { overall_health: UNKNOWN } } }
I20260929 02:49:13.632716 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.626305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.4.197.129:33635)
0929 02:49:13.626504 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.626510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.626664 (+ 154us) server_negotiation.cc:407] Connection header received
0929 02:49:13.626978 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.626983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.627074 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.627202 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:13.627629 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.628421 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.630371 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.630794 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.630902 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.631510 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:13.632116 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:13.632243 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.632372 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.632462 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:49:13.632863 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.626312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:33635 (local address 127.0.0.1:39044)
0929 02:49:13.626574 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.626594 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.626757 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.627254 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.627259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.627273 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:49:13.627494 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.627506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.628553 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.628559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.630212 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.630229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.630998 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.631006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.631220 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.631303 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:49:13.632224 (+ 921us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:49:13.632251 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.632343 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.632520 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 02:49:13.644104 7072 raft_consensus.cc:1261] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Refusing update from remote peer 79e14a4d103f466e9ce863a58daeb3ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:13.644162 6933 raft_consensus.cc:1261] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Refusing update from remote peer 79e14a4d103f466e9ce863a58daeb3ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:13.645087 7147 consensus_queue.cc:1035] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:13.645784 7147 consensus_queue.cc:1035] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:13.656896 7148 tablet_replica.cc:439] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: TxnStatusTablet state changed. Reason: New leader 79e14a4d103f466e9ce863a58daeb3ac. Latest consensus state: current_term: 1 leader_uuid: "79e14a4d103f466e9ce863a58daeb3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } }
I20260929 02:49:13.657215 7148 tablet_replica.cc:442] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:13.662765 7164 tablet_replica.cc:439] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79e14a4d103f466e9ce863a58daeb3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } }
I20260929 02:49:13.663080 7164 tablet_replica.cc:442] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:49:13.665920 7169 mvcc.cc:205] Tried to move back new op lower bound from 7334503029314797568 to 7334503029130543104. Current Snapshot: MvccSnapshot[applied={T|T < 7334503029314797568 or (T in {7334503029314797568})}]
I20260929 02:49:13.669438 7149 tablet_replica.cc:439] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: TxnStatusTablet state changed. Reason: New leader 79e14a4d103f466e9ce863a58daeb3ac. Latest consensus state: current_term: 1 leader_uuid: "79e14a4d103f466e9ce863a58daeb3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } }
I20260929 02:49:13.669713 7149 tablet_replica.cc:442] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:13.669883 7148 tablet_replica.cc:439] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79e14a4d103f466e9ce863a58daeb3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } }
I20260929 02:49:13.670145 7148 tablet_replica.cc:442] T 44a40ff21e8f4331b9ae41f39372bc1f P f694ac5962cb487dbd62d8c42efafe8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:13.670024 7147 tablet_replica.cc:439] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79e14a4d103f466e9ce863a58daeb3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } }
I20260929 02:49:13.670320 7147 tablet_replica.cc:442] T 44a40ff21e8f4331b9ae41f39372bc1f P 79e14a4d103f466e9ce863a58daeb3ac: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:49:13.685037 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.675784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.4.197.131:41139)
0929 02:49:13.675978 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.675985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.677595 (+ 1610us) server_negotiation.cc:407] Connection header received
0929 02:49:13.677817 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.677823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.677927 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.678081 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:13.678718 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.679915 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.682035 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.682685 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.682782 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.683478 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:13.684240 (+ 762us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:13.684364 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.684615 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.684725 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:49:13.685696 7149 tablet_replica.cc:439] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79e14a4d103f466e9ce863a58daeb3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } } }
I20260929 02:49:13.685971 7149 tablet_replica.cc:442] T 44a40ff21e8f4331b9ae41f39372bc1f P 4ccec73cc5dc4c9581797eea02be54c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:13.701682 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.696389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:59087 (local address 127.4.197.129:33635)
0929 02:49:13.696556 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.696561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.697084 (+ 523us) server_negotiation.cc:407] Connection header received
0929 02:49:13.697255 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.697260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.697325 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.697403 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:13.697877 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.698636 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.699975 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.700401 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.700509 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.700952 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:13.701064 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:13.701141 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.701266 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.701340 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:49:13.701723 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.696391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:33635 (local address 127.4.197.131:59087)
0929 02:49:13.696976 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.696992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.697126 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.697424 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.697428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.697440 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:49:13.697718 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.697727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.698759 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.698763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.699836 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.699848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.700568 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.700575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.700688 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.700739 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:49:13.701165 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:49:13.701175 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.701238 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.701377 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":277,"threads_started":1}
I20260929 02:49:13.711283 6933 raft_consensus.cc:1261] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Refusing update from remote peer f694ac5962cb487dbd62d8c42efafe8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:13.711472 6798 raft_consensus.cc:1261] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Refusing update from remote peer f694ac5962cb487dbd62d8c42efafe8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:13.712195 7151 consensus_queue.cc:1035] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:13.712546 7148 consensus_queue.cc:1035] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:13.717162 7170 mvcc.cc:205] Tried to move back new op lower bound from 7334503029594959872 to 7334503028308512768. Current Snapshot: MvccSnapshot[applied={T|T < 7334503029594959872}]
I20260929 02:49:13.733227 7171 mvcc.cc:205] Tried to move back new op lower bound from 7334503029594959872 to 7334503028308512768. Current Snapshot: MvccSnapshot[applied={T|T < 7334503029594959872}]
I20260929 02:49:13.755923 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.748315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:34425 (local address 127.4.197.131:41139)
0929 02:49:13.748498 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.748502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.749118 (+ 616us) server_negotiation.cc:407] Connection header received
0929 02:49:13.749346 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.749352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.749441 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.749533 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:13.750533 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.751705 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.753842 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.754511 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.754640 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.755128 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:13.755267 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:13.755353 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.755551 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.755636 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:49:13.756083 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.748380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:41139 (local address 127.4.197.129:34425)
0929 02:49:13.749007 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.749028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.749180 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.749899 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.749904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.749919 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:49:13.750327 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.750342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.751825 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.751831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.753633 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.753651 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.754706 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.754713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.754849 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.754946 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:49:13.755414 (+ 468us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:49:13.755426 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.755504 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.755662 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":315,"threads_started":1}
I20260929 02:49:13.792520 6635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:13.792894 6635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:13.797286 6635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a: Sending DeleteTablet for 3 replicas of tablet c901ad0a0c8841edbad9facd78ce4e35
I20260929 02:49:13.798687 7052 tablet_service.cc:1514] Processing DeleteTablet for tablet c901ad0a0c8841edbad9facd78ce4e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:13 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:52200
I20260929 02:49:13.800151 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet c901ad0a0c8841edbad9facd78ce4e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:13 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:52012
I20260929 02:49:13.800228 6637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:13.800901 6637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:13.805157 6774 tablet_service.cc:1514] Processing DeleteTablet for tablet c901ad0a0c8841edbad9facd78ce4e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:13 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:39038
I20260929 02:49:13.810542 6773 tablet_service.cc:1467] Processing CreateTablet for tablet cc5e3ff70a0745cb945d313015490afe (DEFAULT_TABLE table=test-table [id=dc1a927d50b743c4bb45b2583629b97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:13.811174 6773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5e3ff70a0745cb945d313015490afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:13.811919 7051 tablet_service.cc:1467] Processing CreateTablet for tablet cc5e3ff70a0745cb945d313015490afe (DEFAULT_TABLE table=test-table [id=dc1a927d50b743c4bb45b2583629b97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:13.812492 7051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5e3ff70a0745cb945d313015490afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:13.817378 7141 tablet_bootstrap.cc:492] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac: Bootstrap starting.
I20260929 02:49:13.817816 6912 tablet_service.cc:1467] Processing CreateTablet for tablet cc5e3ff70a0745cb945d313015490afe (DEFAULT_TABLE table=test-table [id=dc1a927d50b743c4bb45b2583629b97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:13.818342 6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5e3ff70a0745cb945d313015490afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:13.818980 7141 tablet_bootstrap.cc:654] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:13.821981 7142 tablet_bootstrap.cc:492] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a: Bootstrap starting.
I20260929 02:49:13.823712 7142 tablet_bootstrap.cc:654] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:13.824687 7143 tablet_bootstrap.cc:492] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4: Bootstrap starting.
I20260929 02:49:13.826352 7143 tablet_bootstrap.cc:654] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:13.835877 7141 tablet_bootstrap.cc:492] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac: No bootstrap required, opened a new log
I20260929 02:49:13.836088 7141 ts_tablet_manager.cc:1397] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.007s
I20260929 02:49:13.836926 7141 raft_consensus.cc:348] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.837182 7141 raft_consensus.cc:374] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:13.837251 7141 raft_consensus.cc:729] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e14a4d103f466e9ce863a58daeb3ac, State: Initialized, Role: FOLLOWER
I20260929 02:49:13.837529 7141 consensus_queue.cc:260] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.840651 7141 ts_tablet_manager.cc:1428] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260929 02:49:13.843896 7143 tablet_bootstrap.cc:492] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4: No bootstrap required, opened a new log
I20260929 02:49:13.844074 7143 ts_tablet_manager.cc:1397] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.006s
I20260929 02:49:13.844089 7142 tablet_bootstrap.cc:492] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a: No bootstrap required, opened a new log
I20260929 02:49:13.844259 7142 ts_tablet_manager.cc:1397] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.003s
I20260929 02:49:13.844827 7143 raft_consensus.cc:348] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.845080 7143 raft_consensus.cc:374] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:13.844977 7142 raft_consensus.cc:348] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.845152 7143 raft_consensus.cc:729] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ccec73cc5dc4c9581797eea02be54c4, State: Initialized, Role: FOLLOWER
I20260929 02:49:13.845207 7142 raft_consensus.cc:374] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:13.845283 7142 raft_consensus.cc:729] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f694ac5962cb487dbd62d8c42efafe8a, State: Initialized, Role: FOLLOWER
I20260929 02:49:13.845430 7143 consensus_queue.cc:260] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.845537 7142 consensus_queue.cc:260] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:13.846799 7142 ts_tablet_manager.cc:1428] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:13.851641 7143 ts_tablet_manager.cc:1428] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260929 02:49:13.853053 7192 tablet_replica.cc:330] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: stopping tablet replica
I20260929 02:49:13.853562 7192 raft_consensus.cc:2229] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:13.853955 7192 raft_consensus.cc:2258] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:13.854627 7191 tablet_replica.cc:330] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: stopping tablet replica
I20260929 02:49:13.855123 7191 raft_consensus.cc:2229] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:13.855491 7191 raft_consensus.cc:2258] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:13.856869 7190 tablet_replica.cc:330] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: stopping tablet replica
I20260929 02:49:13.857264 7190 raft_consensus.cc:2229] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:13.857396 7192 ts_tablet_manager.cc:1905] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:13.859179 7191 ts_tablet_manager.cc:1905] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:13.859659 7190 raft_consensus.cc:2258] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:13.861505 7190 ts_tablet_manager.cc:1905] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:13.865649 7192 ts_tablet_manager.cc:1918] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:49:13.865828 7192 log.cc:1196] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/c901ad0a0c8841edbad9facd78ce4e35
I20260929 02:49:13.866371 7192 ts_tablet_manager.cc:1939] T c901ad0a0c8841edbad9facd78ce4e35 P 79e14a4d103f466e9ce863a58daeb3ac: Deleting consensus metadata
I20260929 02:49:13.868206 6623 catalog_manager.cc:4841] TS 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129:33635): tablet c901ad0a0c8841edbad9facd78ce4e35 (table test-table [id=43200604f9e946138a061e00917c7bd0]) successfully deleted
I20260929 02:49:13.872097 7190 ts_tablet_manager.cc:1918] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:49:13.872274 7190 log.cc:1196] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/c901ad0a0c8841edbad9facd78ce4e35
I20260929 02:49:13.872778 7190 ts_tablet_manager.cc:1939] T c901ad0a0c8841edbad9facd78ce4e35 P f694ac5962cb487dbd62d8c42efafe8a: Deleting consensus metadata
I20260929 02:49:13.874009 7191 ts_tablet_manager.cc:1918] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:49:13.874173 7191 log.cc:1196] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/c901ad0a0c8841edbad9facd78ce4e35
I20260929 02:49:13.874506 6622 catalog_manager.cc:4841] TS f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139): tablet c901ad0a0c8841edbad9facd78ce4e35 (table test-table [id=43200604f9e946138a061e00917c7bd0]) successfully deleted
I20260929 02:49:13.874984 7191 ts_tablet_manager.cc:1939] T c901ad0a0c8841edbad9facd78ce4e35 P 4ccec73cc5dc4c9581797eea02be54c4: Deleting consensus metadata
I20260929 02:49:13.876298 6624 catalog_manager.cc:4841] TS 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523): tablet c901ad0a0c8841edbad9facd78ce4e35 (table test-table [id=43200604f9e946138a061e00917c7bd0]) successfully deleted
I20260929 02:49:14.134477 7148 raft_consensus.cc:482] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:14.134708 7148 raft_consensus.cc:504] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:14.135596 7148 leader_election.cc:288] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129:33635), 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523)
I20260929 02:49:14.136077 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5e3ff70a0745cb945d313015490afe" candidate_uuid: "f694ac5962cb487dbd62d8c42efafe8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e14a4d103f466e9ce863a58daeb3ac" is_pre_election: true
I20260929 02:49:14.136309 6798 raft_consensus.cc:2454] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f694ac5962cb487dbd62d8c42efafe8a in term 0.
I20260929 02:49:14.136339 6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5e3ff70a0745cb945d313015490afe" candidate_uuid: "f694ac5962cb487dbd62d8c42efafe8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ccec73cc5dc4c9581797eea02be54c4" is_pre_election: true
I20260929 02:49:14.136597 6932 raft_consensus.cc:2454] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f694ac5962cb487dbd62d8c42efafe8a in term 0.
I20260929 02:49:14.137101 7005 leader_election.cc:302] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79e14a4d103f466e9ce863a58daeb3ac, f694ac5962cb487dbd62d8c42efafe8a; no voters:
I20260929 02:49:14.137430 7148 raft_consensus.cc:2781] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:14.137549 7148 raft_consensus.cc:482] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:14.137634 7148 raft_consensus.cc:3037] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:14.140923 7148 raft_consensus.cc:504] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:14.141615 7148 leader_election.cc:288] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [CANDIDATE]: Term 1 election: Requested vote from peers 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129:33635), 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523)
I20260929 02:49:14.142208 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5e3ff70a0745cb945d313015490afe" candidate_uuid: "f694ac5962cb487dbd62d8c42efafe8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e14a4d103f466e9ce863a58daeb3ac"
I20260929 02:49:14.142376 6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5e3ff70a0745cb945d313015490afe" candidate_uuid: "f694ac5962cb487dbd62d8c42efafe8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ccec73cc5dc4c9581797eea02be54c4"
I20260929 02:49:14.142464 6798 raft_consensus.cc:3037] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:14.142606 6932 raft_consensus.cc:3037] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:14.145582 6798 raft_consensus.cc:2454] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f694ac5962cb487dbd62d8c42efafe8a in term 1.
I20260929 02:49:14.145882 6932 raft_consensus.cc:2454] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f694ac5962cb487dbd62d8c42efafe8a in term 1.
I20260929 02:49:14.146481 7006 leader_election.cc:302] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ccec73cc5dc4c9581797eea02be54c4, f694ac5962cb487dbd62d8c42efafe8a; no voters:
I20260929 02:49:14.146952 7148 raft_consensus.cc:2781] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:14.147128 7148 raft_consensus.cc:686] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 1 LEADER]: Becoming Leader. State: Replica: f694ac5962cb487dbd62d8c42efafe8a, State: Running, Role: LEADER
I20260929 02:49:14.147420 7148 consensus_queue.cc:237] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:14.152089 6635 catalog_manager.cc:5495] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a reported cstate change: term changed from 0 to 1, leader changed from <none> to f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131). New cstate: current_term: 1 leader_uuid: "f694ac5962cb487dbd62d8c42efafe8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } health_report { overall_health: HEALTHY } } }
I20260929 02:49:14.203138 6932 raft_consensus.cc:1261] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Refusing update from remote peer f694ac5962cb487dbd62d8c42efafe8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:14.203414 6798 raft_consensus.cc:1261] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Refusing update from remote peer f694ac5962cb487dbd62d8c42efafe8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:14.203635 7148 consensus_queue.cc:1035] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:14.203944 7185 consensus_queue.cc:1035] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:49:14.226351 7170 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 02:49:15.807184 6636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:15.807384 6636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:15.811136 6636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a: Sending DeleteTablet for 3 replicas of tablet cc5e3ff70a0745cb945d313015490afe
I20260929 02:49:15.811874 6774 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5e3ff70a0745cb945d313015490afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:15 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:39038
I20260929 02:49:15.811920 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5e3ff70a0745cb945d313015490afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:15 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:52012
I20260929 02:49:15.812144 7052 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5e3ff70a0745cb945d313015490afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:15 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:52200
I20260929 02:49:15.823108 7223 tablet_replica.cc:330] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a: stopping tablet replica
I20260929 02:49:15.823385 7223 raft_consensus.cc:2229] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:15.823808 7223 raft_consensus.cc:2258] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:15.824373 7221 tablet_replica.cc:330] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac: stopping tablet replica
I20260929 02:49:15.824424 7222 tablet_replica.cc:330] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4: stopping tablet replica
I20260929 02:49:15.824682 7221 raft_consensus.cc:2229] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:15.824766 7222 raft_consensus.cc:2229] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:15.824939 7221 raft_consensus.cc:2258] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:15.824999 7222 raft_consensus.cc:2258] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:15.825939 7222 ts_tablet_manager.cc:1905] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:15.826184 7221 ts_tablet_manager.cc:1905] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:15.830241 7223 ts_tablet_manager.cc:1905] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:15.832605 7222 ts_tablet_manager.cc:1918] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:49:15.832607 7221 ts_tablet_manager.cc:1918] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:49:15.832762 7222 log.cc:1196] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/cc5e3ff70a0745cb945d313015490afe
I20260929 02:49:15.832823 7221 log.cc:1196] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/cc5e3ff70a0745cb945d313015490afe
I20260929 02:49:15.833235 7222 ts_tablet_manager.cc:1939] T cc5e3ff70a0745cb945d313015490afe P 4ccec73cc5dc4c9581797eea02be54c4: Deleting consensus metadata
I20260929 02:49:15.833241 7221 ts_tablet_manager.cc:1939] T cc5e3ff70a0745cb945d313015490afe P 79e14a4d103f466e9ce863a58daeb3ac: Deleting consensus metadata
I20260929 02:49:15.834431 6623 catalog_manager.cc:4841] TS 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129:33635): tablet cc5e3ff70a0745cb945d313015490afe (table test-table [id=dc1a927d50b743c4bb45b2583629b97a]) successfully deleted
I20260929 02:49:15.834623 6624 catalog_manager.cc:4841] TS 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523): tablet cc5e3ff70a0745cb945d313015490afe (table test-table [id=dc1a927d50b743c4bb45b2583629b97a]) successfully deleted
I20260929 02:49:15.837235 7223 ts_tablet_manager.cc:1918] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:49:15.837422 7223 log.cc:1196] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/cc5e3ff70a0745cb945d313015490afe
I20260929 02:49:15.837780 7223 ts_tablet_manager.cc:1939] T cc5e3ff70a0745cb945d313015490afe P f694ac5962cb487dbd62d8c42efafe8a: Deleting consensus metadata
I20260929 02:49:15.838866 6622 catalog_manager.cc:4841] TS f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139): tablet cc5e3ff70a0745cb945d313015490afe (table test-table [id=dc1a927d50b743c4bb45b2583629b97a]) successfully deleted
Sep 29 02:49:15 dist-test-slave-t6c1 krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:49:15 dist-test-slave-t6c1 krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:15.869755 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.853279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.4.197.190:36831)
0929 02:49:15.853792 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.853797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.853824 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:49:15.853983 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.853987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.854047 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.854145 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.855485 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.856232 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.857472 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.857941 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.858054 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.864562 (+ 6508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.864568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.864571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.864606 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.867058 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.867790 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.867796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.867798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.867940 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.868313 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.868317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.868319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.868627 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.868857 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.869154 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.869399 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":278,"threads_started":1}
I20260929 02:49:15.874136 6636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:15.874766 6636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:15.881413 6774 tablet_service.cc:1467] Processing CreateTablet for tablet 9d18ea04493f476d89b22be3848e142c (DEFAULT_TABLE table=test-table [id=0e3e0f2c382f4a7ea067d89e872ece1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:15.881778 7052 tablet_service.cc:1467] Processing CreateTablet for tablet 9d18ea04493f476d89b22be3848e142c (DEFAULT_TABLE table=test-table [id=0e3e0f2c382f4a7ea067d89e872ece1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:15.881829 6913 tablet_service.cc:1467] Processing CreateTablet for tablet 9d18ea04493f476d89b22be3848e142c (DEFAULT_TABLE table=test-table [id=0e3e0f2c382f4a7ea067d89e872ece1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:15.881930 6774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d18ea04493f476d89b22be3848e142c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:15.882203 7052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d18ea04493f476d89b22be3848e142c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:15.882232 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d18ea04493f476d89b22be3848e142c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:15.887300 7232 tablet_bootstrap.cc:492] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4: Bootstrap starting.
I20260929 02:49:15.889545 7232 tablet_bootstrap.cc:654] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:15.889605 7234 tablet_bootstrap.cc:492] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a: Bootstrap starting.
I20260929 02:49:15.890952 7233 tablet_bootstrap.cc:492] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac: Bootstrap starting.
I20260929 02:49:15.891390 7234 tablet_bootstrap.cc:654] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:15.892526 7233 tablet_bootstrap.cc:654] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:15.893024 7232 tablet_bootstrap.cc:492] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4: No bootstrap required, opened a new log
I20260929 02:49:15.893173 7232 ts_tablet_manager.cc:1397] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 02:49:15.894068 7234 tablet_bootstrap.cc:492] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a: No bootstrap required, opened a new log
I20260929 02:49:15.893970 7232 raft_consensus.cc:348] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:15.894189 7232 raft_consensus.cc:374] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:15.894199 7234 ts_tablet_manager.cc:1397] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:49:15.894284 7232 raft_consensus.cc:729] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ccec73cc5dc4c9581797eea02be54c4, State: Initialized, Role: FOLLOWER
I20260929 02:49:15.894338 7233 tablet_bootstrap.cc:492] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac: No bootstrap required, opened a new log
I20260929 02:49:15.894467 7233 ts_tablet_manager.cc:1397] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:49:15.894552 7232 consensus_queue.cc:260] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:15.894951 7234 raft_consensus.cc:348] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:15.895202 7234 raft_consensus.cc:374] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:15.895203 7233 raft_consensus.cc:348] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:15.895485 7233 raft_consensus.cc:374] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:15.895509 7234 raft_consensus.cc:729] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f694ac5962cb487dbd62d8c42efafe8a, State: Initialized, Role: FOLLOWER
I20260929 02:49:15.895565 7233 raft_consensus.cc:729] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e14a4d103f466e9ce863a58daeb3ac, State: Initialized, Role: FOLLOWER
I20260929 02:49:15.895772 7234 consensus_queue.cc:260] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:15.895807 7233 consensus_queue.cc:260] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:15.896344 7233 ts_tablet_manager.cc:1428] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:15.896870 7234 ts_tablet_manager.cc:1428] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:49:15.897097 7232 ts_tablet_manager.cc:1428] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260929 02:49:16.142679 7207 raft_consensus.cc:482] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:16.142938 7207 raft_consensus.cc:504] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.143688 7207 leader_election.cc:288] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523), f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139)
I20260929 02:49:16.144335 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d18ea04493f476d89b22be3848e142c" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ccec73cc5dc4c9581797eea02be54c4" is_pre_election: true
I20260929 02:49:16.144339 7072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d18ea04493f476d89b22be3848e142c" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f694ac5962cb487dbd62d8c42efafe8a" is_pre_election: true
I20260929 02:49:16.144562 6933 raft_consensus.cc:2454] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 0.
I20260929 02:49:16.144562 7072 raft_consensus.cc:2454] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 0.
I20260929 02:49:16.145033 6725 leader_election.cc:302] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79e14a4d103f466e9ce863a58daeb3ac, f694ac5962cb487dbd62d8c42efafe8a; no voters:
I20260929 02:49:16.145354 7207 raft_consensus.cc:2781] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:16.145475 7207 raft_consensus.cc:482] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:16.145541 7207 raft_consensus.cc:3037] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:16.148592 7207 raft_consensus.cc:504] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.149271 7207 leader_election.cc:288] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 election: Requested vote from peers 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523), f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139)
I20260929 02:49:16.149894 7072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d18ea04493f476d89b22be3848e142c" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f694ac5962cb487dbd62d8c42efafe8a"
I20260929 02:49:16.149897 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d18ea04493f476d89b22be3848e142c" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ccec73cc5dc4c9581797eea02be54c4"
I20260929 02:49:16.150120 7072 raft_consensus.cc:3037] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:16.150126 6933 raft_consensus.cc:3037] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:16.152540 6933 raft_consensus.cc:2454] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 1.
I20260929 02:49:16.152994 6727 leader_election.cc:302] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ccec73cc5dc4c9581797eea02be54c4, 79e14a4d103f466e9ce863a58daeb3ac; no voters:
I20260929 02:49:16.153049 7072 raft_consensus.cc:2454] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 1.
I20260929 02:49:16.153322 7207 raft_consensus.cc:2781] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:16.153512 7207 raft_consensus.cc:686] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 1 LEADER]: Becoming Leader. State: Replica: 79e14a4d103f466e9ce863a58daeb3ac, State: Running, Role: LEADER
I20260929 02:49:16.153826 7207 consensus_queue.cc:237] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.156621 6635 catalog_manager.cc:5495] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129). New cstate: current_term: 1 leader_uuid: "79e14a4d103f466e9ce863a58daeb3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } health_report { overall_health: UNKNOWN } } }
I20260929 02:49:16.197789 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:16.193249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.4.197.129:33635)
0929 02:49:16.193699 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:16.193703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:16.193721 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:49:16.193769 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:16.193772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:16.193830 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:16.193942 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:16.194302 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:16.195110 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:16.196373 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:16.196787 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:16.196884 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:16.197220 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:16.197323 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:16.197390 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:16.197477 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 02:49:16.197533 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260929 02:49:16.210579 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:16.205513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:54621 (local address 127.4.197.129:33635)
0929 02:49:16.205645 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:16.205649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:16.206480 (+ 831us) server_negotiation.cc:407] Connection header received
0929 02:49:16.206629 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:16.206632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:16.206681 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:16.206747 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:16.207188 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:16.207848 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:16.209117 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:16.209509 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:16.209606 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:16.209933 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:16.210020 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:16.210090 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:16.210212 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 02:49:16.210273 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 02:49:16.210636 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:16.205517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:33635 (local address 127.4.197.129:54621)
0929 02:49:16.206392 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:16.206407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:16.206529 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:16.206762 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:16.206766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:16.206776 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:49:16.207052 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:16.207062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:16.207986 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:16.207992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:16.208997 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:16.209008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:16.209644 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:16.209650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:16.209772 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:16.209802 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:49:16.210118 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:49:16.210127 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:49:16.210192 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 02:49:16.210309 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":257,"threads_started":1}
I20260929 02:49:16.217784 7072 raft_consensus.cc:1261] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Refusing update from remote peer 79e14a4d103f466e9ce863a58daeb3ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:16.217809 6933 raft_consensus.cc:1261] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Refusing update from remote peer 79e14a4d103f466e9ce863a58daeb3ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:16.218221 7207 consensus_queue.cc:1035] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:16.218451 7201 consensus_queue.cc:1035] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:16.266455 6635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:16.266726 6635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:16.269743 6635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a: Sending DeleteTablet for 3 replicas of tablet 9d18ea04493f476d89b22be3848e142c
I20260929 02:49:16.270503 6773 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d18ea04493f476d89b22be3848e142c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:16 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:39038
I20260929 02:49:16.270630 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d18ea04493f476d89b22be3848e142c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:16 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:52012
I20260929 02:49:16.270776 7221 tablet_replica.cc:330] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac: stopping tablet replica
I20260929 02:49:16.270988 7221 raft_consensus.cc:2229] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:16.271057 7222 tablet_replica.cc:330] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4: stopping tablet replica
I20260929 02:49:16.271247 7222 raft_consensus.cc:2229] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:16.271432 7222 raft_consensus.cc:2258] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:16.271494 7221 raft_consensus.cc:2258] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:16.271802 7052 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d18ea04493f476d89b22be3848e142c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:16 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:52200
I20260929 02:49:16.272095 7223 tablet_replica.cc:330] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a: stopping tablet replica
I20260929 02:49:16.272279 7223 raft_consensus.cc:2229] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:16.272466 7223 raft_consensus.cc:2258] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:16.272622 7221 ts_tablet_manager.cc:1905] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:16.272501 6635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:16.273370 6635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:16.273661 7223 ts_tablet_manager.cc:1905] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:16.279603 7221 ts_tablet_manager.cc:1918] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:49:16.279780 7221 log.cc:1196] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/9d18ea04493f476d89b22be3848e142c
I20260929 02:49:16.280202 7221 ts_tablet_manager.cc:1939] T 9d18ea04493f476d89b22be3848e142c P 79e14a4d103f466e9ce863a58daeb3ac: Deleting consensus metadata
I20260929 02:49:16.280884 6913 tablet_service.cc:1467] Processing CreateTablet for tablet aa5fc112d00541d1a018b6b7794d5f31 (DEFAULT_TABLE table=test-table [id=ea1a194f3a9d4cfc963929201a65826a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:16.281383 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5fc112d00541d1a018b6b7794d5f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:16.281744 6623 catalog_manager.cc:4841] TS 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129:33635): tablet 9d18ea04493f476d89b22be3848e142c (table test-table [id=0e3e0f2c382f4a7ea067d89e872ece1e]) successfully deleted
I20260929 02:49:16.281775 6773 tablet_service.cc:1467] Processing CreateTablet for tablet aa5fc112d00541d1a018b6b7794d5f31 (DEFAULT_TABLE table=test-table [id=ea1a194f3a9d4cfc963929201a65826a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:16.282167 6773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5fc112d00541d1a018b6b7794d5f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:16.283277 7052 tablet_service.cc:1467] Processing CreateTablet for tablet aa5fc112d00541d1a018b6b7794d5f31 (DEFAULT_TABLE table=test-table [id=ea1a194f3a9d4cfc963929201a65826a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:16.283682 7052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5fc112d00541d1a018b6b7794d5f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:16.285121 7223 ts_tablet_manager.cc:1918] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:49:16.286402 7223 log.cc:1196] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/9d18ea04493f476d89b22be3848e142c
I20260929 02:49:16.286952 7223 ts_tablet_manager.cc:1939] T 9d18ea04493f476d89b22be3848e142c P f694ac5962cb487dbd62d8c42efafe8a: Deleting consensus metadata
I20260929 02:49:16.288089 6622 catalog_manager.cc:4841] TS f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139): tablet 9d18ea04493f476d89b22be3848e142c (table test-table [id=0e3e0f2c382f4a7ea067d89e872ece1e]) successfully deleted
I20260929 02:49:16.288430 7233 tablet_bootstrap.cc:492] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac: Bootstrap starting.
I20260929 02:49:16.289894 7222 ts_tablet_manager.cc:1905] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:16.290652 7233 tablet_bootstrap.cc:654] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:16.296177 7222 ts_tablet_manager.cc:1918] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:49:16.296337 7222 log.cc:1196] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/9d18ea04493f476d89b22be3848e142c
I20260929 02:49:16.296711 7222 ts_tablet_manager.cc:1939] T 9d18ea04493f476d89b22be3848e142c P 4ccec73cc5dc4c9581797eea02be54c4: Deleting consensus metadata
I20260929 02:49:16.297729 6624 catalog_manager.cc:4841] TS 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523): tablet 9d18ea04493f476d89b22be3848e142c (table test-table [id=0e3e0f2c382f4a7ea067d89e872ece1e]) successfully deleted
I20260929 02:49:16.299026 7234 tablet_bootstrap.cc:492] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a: Bootstrap starting.
I20260929 02:49:16.300335 7232 tablet_bootstrap.cc:492] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4: Bootstrap starting.
I20260929 02:49:16.300590 7234 tablet_bootstrap.cc:654] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:16.301894 7232 tablet_bootstrap.cc:654] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:16.304741 7232 tablet_bootstrap.cc:492] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4: No bootstrap required, opened a new log
I20260929 02:49:16.304883 7232 ts_tablet_manager.cc:1397] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 02:49:16.305464 7234 tablet_bootstrap.cc:492] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a: No bootstrap required, opened a new log
I20260929 02:49:16.305411 7232 raft_consensus.cc:348] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.305610 7232 raft_consensus.cc:374] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:16.305632 7234 ts_tablet_manager.cc:1397] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260929 02:49:16.305687 7232 raft_consensus.cc:729] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ccec73cc5dc4c9581797eea02be54c4, State: Initialized, Role: FOLLOWER
I20260929 02:49:16.305982 7233 tablet_bootstrap.cc:492] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac: No bootstrap required, opened a new log
I20260929 02:49:16.305929 7232 consensus_queue.cc:260] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.306252 7233 ts_tablet_manager.cc:1397] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.014s
I20260929 02:49:16.306406 7232 ts_tablet_manager.cc:1428] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 02:49:16.306351 7234 raft_consensus.cc:348] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.306597 7234 raft_consensus.cc:374] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:16.306674 7234 raft_consensus.cc:729] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f694ac5962cb487dbd62d8c42efafe8a, State: Initialized, Role: FOLLOWER
I20260929 02:49:16.306892 7234 consensus_queue.cc:260] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.307430 7234 ts_tablet_manager.cc:1428] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:49:16.307358 7233 raft_consensus.cc:348] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.307693 7233 raft_consensus.cc:374] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:16.307825 7233 raft_consensus.cc:729] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e14a4d103f466e9ce863a58daeb3ac, State: Initialized, Role: FOLLOWER
I20260929 02:49:16.308112 7233 consensus_queue.cc:260] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.308736 7233 ts_tablet_manager.cc:1428] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:16.522665 7202 raft_consensus.cc:482] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:16.522912 7202 raft_consensus.cc:504] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.523918 7202 leader_election.cc:288] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523), f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139)
I20260929 02:49:16.524456 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5fc112d00541d1a018b6b7794d5f31" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ccec73cc5dc4c9581797eea02be54c4" is_pre_election: true
I20260929 02:49:16.524463 7071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5fc112d00541d1a018b6b7794d5f31" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f694ac5962cb487dbd62d8c42efafe8a" is_pre_election: true
I20260929 02:49:16.524673 6933 raft_consensus.cc:2454] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 0.
I20260929 02:49:16.524708 7071 raft_consensus.cc:2454] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 0.
I20260929 02:49:16.525094 6727 leader_election.cc:302] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ccec73cc5dc4c9581797eea02be54c4, 79e14a4d103f466e9ce863a58daeb3ac; no voters:
I20260929 02:49:16.525395 7202 raft_consensus.cc:2781] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:16.525521 7202 raft_consensus.cc:482] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:16.525607 7202 raft_consensus.cc:3037] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:16.528208 7202 raft_consensus.cc:504] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.528928 7202 leader_election.cc:288] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 election: Requested vote from peers 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523), f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139)
I20260929 02:49:16.529492 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5fc112d00541d1a018b6b7794d5f31" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ccec73cc5dc4c9581797eea02be54c4"
I20260929 02:49:16.529734 6933 raft_consensus.cc:3037] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:16.529994 7071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5fc112d00541d1a018b6b7794d5f31" candidate_uuid: "79e14a4d103f466e9ce863a58daeb3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f694ac5962cb487dbd62d8c42efafe8a"
I20260929 02:49:16.530203 7071 raft_consensus.cc:3037] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:16.531905 6933 raft_consensus.cc:2454] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 1.
I20260929 02:49:16.532320 6727 leader_election.cc:302] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ccec73cc5dc4c9581797eea02be54c4, 79e14a4d103f466e9ce863a58daeb3ac; no voters:
I20260929 02:49:16.532671 7202 raft_consensus.cc:2781] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:16.532825 7202 raft_consensus.cc:686] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 1 LEADER]: Becoming Leader. State: Replica: 79e14a4d103f466e9ce863a58daeb3ac, State: Running, Role: LEADER
I20260929 02:49:16.533098 7202 consensus_queue.cc:237] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } }
I20260929 02:49:16.533284 7071 raft_consensus.cc:2454] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e14a4d103f466e9ce863a58daeb3ac in term 1.
I20260929 02:49:16.536167 6636 catalog_manager.cc:5495] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129). New cstate: current_term: 1 leader_uuid: "79e14a4d103f466e9ce863a58daeb3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e14a4d103f466e9ce863a58daeb3ac" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 } health_report { overall_health: UNKNOWN } } }
I20260929 02:49:16.607607 7071 raft_consensus.cc:1261] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Refusing update from remote peer 79e14a4d103f466e9ce863a58daeb3ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:16.607628 6933 raft_consensus.cc:1261] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Refusing update from remote peer 79e14a4d103f466e9ce863a58daeb3ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:16.608203 7201 consensus_queue.cc:1035] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f694ac5962cb487dbd62d8c42efafe8a" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:16.608460 7247 consensus_queue.cc:1035] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ccec73cc5dc4c9581797eea02be54c4" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:49:16.631063 7199 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 02:49:18.216238 6636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:18.216432 6636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:18.220008 6636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 419c732062fc49559fa22b96815f628a: Sending DeleteTablet for 3 replicas of tablet aa5fc112d00541d1a018b6b7794d5f31
I20260929 02:49:18.220625 6773 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5fc112d00541d1a018b6b7794d5f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:18 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:39038
I20260929 02:49:18.220790 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5fc112d00541d1a018b6b7794d5f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:18 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:52012
I20260929 02:49:18.220870 7052 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5fc112d00541d1a018b6b7794d5f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:18 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:52200
I20260929 02:49:18.221267 7274 tablet_replica.cc:330] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac: stopping tablet replica
I20260929 02:49:18.221453 7275 tablet_replica.cc:330] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4: stopping tablet replica
I20260929 02:49:18.221496 7274 raft_consensus.cc:2229] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:18.221653 7276 tablet_replica.cc:330] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a: stopping tablet replica
I20260929 02:49:18.221684 7275 raft_consensus.cc:2229] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:18.221828 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6708
I20260929 02:49:18.221940 7275 raft_consensus.cc:2258] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:18.221937 7276 raft_consensus.cc:2229] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:18.222144 7276 raft_consensus.cc:2258] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:18.222908 7275 ts_tablet_manager.cc:1905] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:18.223492 7276 ts_tablet_manager.cc:1905] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:18.227207 7274 raft_consensus.cc:2258] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:18.228231 7274 ts_tablet_manager.cc:1905] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:18.229440 7275 ts_tablet_manager.cc:1918] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:49:18.229574 7275 log.cc:1196] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/aa5fc112d00541d1a018b6b7794d5f31
I20260929 02:49:18.230008 7275 ts_tablet_manager.cc:1939] T aa5fc112d00541d1a018b6b7794d5f31 P 4ccec73cc5dc4c9581797eea02be54c4: Deleting consensus metadata
I20260929 02:49:18.231148 6624 catalog_manager.cc:4841] TS 4ccec73cc5dc4c9581797eea02be54c4 (127.4.197.130:42523): tablet aa5fc112d00541d1a018b6b7794d5f31 (table test-table [id=ea1a194f3a9d4cfc963929201a65826a]) successfully deleted
I20260929 02:49:18.232710 7276 ts_tablet_manager.cc:1918] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:49:18.232861 7276 log.cc:1196] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/aa5fc112d00541d1a018b6b7794d5f31
I20260929 02:49:18.233281 7276 ts_tablet_manager.cc:1939] T aa5fc112d00541d1a018b6b7794d5f31 P f694ac5962cb487dbd62d8c42efafe8a: Deleting consensus metadata
I20260929 02:49:18.233444 7274 ts_tablet_manager.cc:1918] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:49:18.233592 7274 log.cc:1196] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/aa5fc112d00541d1a018b6b7794d5f31
I20260929 02:49:18.234038 7274 ts_tablet_manager.cc:1939] T aa5fc112d00541d1a018b6b7794d5f31 P 79e14a4d103f466e9ce863a58daeb3ac: Deleting consensus metadata
I20260929 02:49:18.234440 6622 catalog_manager.cc:4841] TS f694ac5962cb487dbd62d8c42efafe8a (127.4.197.131:41139): tablet aa5fc112d00541d1a018b6b7794d5f31 (table test-table [id=ea1a194f3a9d4cfc963929201a65826a]) successfully deleted
I20260929 02:49:18.235081 6623 catalog_manager.cc:4841] TS 79e14a4d103f466e9ce863a58daeb3ac (127.4.197.129:33635): tablet aa5fc112d00541d1a018b6b7794d5f31 (table test-table [id=ea1a194f3a9d4cfc963929201a65826a]) successfully deleted
Sep 29 02:49:18 dist-test-slave-t6c1 krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:18.244139 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:18.223072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.4.197.129:33635)
0929 02:49:18.227782 (+ 4710us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:18.227787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:18.228226 (+ 439us) server_negotiation.cc:407] Connection header received
0929 02:49:18.228450 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:18.228455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:18.228526 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:18.228635 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:18.230203 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:18.231350 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:18.233155 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:18.233823 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:18.233953 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:18.238123 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:18.238139 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:18.238156 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:18.238198 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:18.241532 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:18.242249 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:18.242258 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:18.242260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:18.242419 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:18.242777 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:18.242780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:18.242782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:18.243172 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:18.243394 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:18.243668 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 02:49:18.243842 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4603,"thread_start_us":4501,"threads_started":1}
W20260929 02:49:18.244788 6839 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41048
W20260929 02:49:18.245398 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6708: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:49:18.245589 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 6708
I20260929 02:49:18.265913 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6848
Sep 29 02:49:18 dist-test-slave-t6c1 krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:49:18.280191 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:18.266629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.4.197.130:42523)
0929 02:49:18.267161 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:18.267166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:18.267182 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:49:18.267238 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:18.267242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:18.267309 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:18.267402 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:18.268669 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:18.269458 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:18.270641 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:18.271208 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:18.271314 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:18.275323 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:18.275338 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:18.275354 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:18.275393 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:18.277937 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:18.278561 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:18.278568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:18.278570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:18.278699 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:18.278989 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:18.278992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:18.278994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:18.279313 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:18.279531 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:18.279743 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 02:49:18.279916 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":297,"threads_started":1}
W20260929 02:49:18.280949 6978 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45802
W20260929 02:49:18.281534 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6848: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:49:18.281663 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 6848
I20260929 02:49:18.295176 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6987
Sep 29 02:49:18 dist-test-slave-t6c1 krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:49:18.309583 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:18.295904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.4.197.131:41139)
0929 02:49:18.296419 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:18.296423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:18.296439 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:49:18.296493 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:18.296497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:18.296561 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:18.296650 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:18.297808 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:18.298508 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:18.299749 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:18.300189 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:18.300307 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:18.304705 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:18.304720 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:18.304735 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:18.304768 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:18.307345 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:18.307977 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:18.307986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:18.307990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:18.308112 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:18.308413 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:18.308416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:18.308418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:18.308741 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:18.308955 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:18.309133 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 02:49:18.309307 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":262,"threads_started":1}
W20260929 02:49:18.310163 7117 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37244
W20260929 02:49:18.310761 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6987: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:49:18.310901 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 6987
I20260929 02:49:18.327662 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 6605
I20260929 02:49:18.328866 6677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:18.445178 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 6605
2026-09-29T02:49:18Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8437 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:18 dist-test-slave-t6c1 krb5kdc[7284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:18 dist-test-slave-t6c1 krb5kdc[7284](info): set up 2 sockets
Sep 29 02:49:18 dist-test-slave-t6c1 krb5kdc[7284](info): commencing operation
krb5kdc: starting...
W20260929 02:49:19.443181 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:49:19 dist-test-slave-t6c1 krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:49:19.615760 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:38207
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:38207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:19.937891 7297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:19.938153 7297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:19.938279 7297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:19.945394 7297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:19.945489 7297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:19.945542 7297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:19.945617 7297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:19.945667 7297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:19.955265 7297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:38207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:38207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:19.956624 7297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:19.957988 7297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:19.969937 7305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:19.969882 7303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:19.970000 7302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:19.970378 7297 server_base.cc:1034] running on GCE node
I20260929 02:49:19.971369 7297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:19.972790 7297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:19.974010 7297 hybrid_clock.cc:648] HybridClock initialized: now 1790650159973948 us; error 47 us; skew 500 ppm
I20260929 02:49:19.974346 7297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:19 dist-test-slave-t6c1 krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:19.977341 7297 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:49:19.978734 7297 webserver.cc:460] Webserver started at http://127.4.197.190:33255/ using document root <none> and password file <none>
I20260929 02:49:19.979465 7297 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:19.979580 7297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:19.979872 7297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:19.982816 7297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "cde46ea2a516411583cc325383ddd126"
format_stamp: "Formatted at 2026-09-29 02:49:19 on dist-test-slave-t6c1"
server_key: "06837d1c3df68c9f9f9f82e8fe51a4c3"
server_key_iv: "f8805ebdce7465bac5e780855a4e0b90"
server_key_version: "encryptionkey@0"
I20260929 02:49:19.983667 7297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "cde46ea2a516411583cc325383ddd126"
format_stamp: "Formatted at 2026-09-29 02:49:19 on dist-test-slave-t6c1"
server_key: "06837d1c3df68c9f9f9f82e8fe51a4c3"
server_key_iv: "f8805ebdce7465bac5e780855a4e0b90"
server_key_version: "encryptionkey@0"
I20260929 02:49:19.988726 7297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:49:19.992156 7312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:19.993463 7297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:19.993621 7297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "cde46ea2a516411583cc325383ddd126"
format_stamp: "Formatted at 2026-09-29 02:49:19 on dist-test-slave-t6c1"
server_key: "06837d1c3df68c9f9f9f82e8fe51a4c3"
server_key_iv: "f8805ebdce7465bac5e780855a4e0b90"
server_key_version: "encryptionkey@0"
I20260929 02:49:19.993803 7297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:20.029086 7297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:20.033455 7297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:20.033736 7297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:20.050294 7297 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:38207
I20260929 02:49:20.050314 7363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:38207 every 8 connection(s)
I20260929 02:49:20.051708 7297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:20.055451 7364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:20.061796 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 7297
I20260929 02:49:20.062047 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:20.062865 4886 external_mini_cluster.cc:1428] Setting key 2ca9573617dca6b5b5b5a8c2d47b8ee9
I20260929 02:49:20.067422 7364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126: Bootstrap starting.
I20260929 02:49:20.070677 7364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:20.071777 7364 log.cc:824] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:20.074564 7364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126: No bootstrap required, opened a new log
Sep 29 02:49:20 dist-test-slave-t6c1 krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:20.083513 7364 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde46ea2a516411583cc325383ddd126" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38207 } }
I20260929 02:49:20.083892 7364 raft_consensus.cc:374] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:20.084002 7364 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde46ea2a516411583cc325383ddd126, State: Initialized, Role: FOLLOWER
I20260929 02:49:20.084715 7364 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde46ea2a516411583cc325383ddd126" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38207 } }
I20260929 02:49:20.085017 7364 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:20.085222 7364 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:20.085400 7364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:20.086290 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:20.065283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.4.197.190:38207)
0929 02:49:20.066124 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:20.066137 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:20.066180 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:49:20.068055 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:20.068083 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:20.068527 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:20.068948 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:20.070399 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:20.071837 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:20.073096 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:20.073787 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:20.073967 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:20.078707 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:20.078715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:20.078732 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:20.078784 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:20.081896 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:20.082727 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:20.082737 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:20.082747 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:20.082981 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:20.083374 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:20.083377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:20.083379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:20.084015 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:20.084257 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:20.084709 (+ 452us) server_negotiation.cc:299] Negotiation successful
0929 02:49:20.085091 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":343,"threads_started":1}
I20260929 02:49:20.087818 7364 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde46ea2a516411583cc325383ddd126" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38207 } }
I20260929 02:49:20.088299 7364 leader_election.cc:302] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde46ea2a516411583cc325383ddd126; no voters:
I20260929 02:49:20.088886 7364 leader_election.cc:288] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:20.089021 7369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:20.089577 7369 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [term 1 LEADER]: Becoming Leader. State: Replica: cde46ea2a516411583cc325383ddd126, State: Running, Role: LEADER
I20260929 02:49:20.090162 7369 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde46ea2a516411583cc325383ddd126" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38207 } }
I20260929 02:49:20.091120 7364 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:20.092818 7371 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde46ea2a516411583cc325383ddd126. Latest consensus state: current_term: 1 leader_uuid: "cde46ea2a516411583cc325383ddd126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde46ea2a516411583cc325383ddd126" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38207 } } }
I20260929 02:49:20.093168 7371 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:20.093329 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cde46ea2a516411583cc325383ddd126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde46ea2a516411583cc325383ddd126" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38207 } } }
I20260929 02:49:20.095484 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:20.096719 7375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:20.101538 7375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:20.109859 7375 catalog_manager.cc:1347] Generated new cluster ID: 2076066e1c0f4686ad138add315a356d
I20260929 02:49:20.109956 7375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:20.140829 7375 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:20.142309 7375 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:20.153555 7375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cde46ea2a516411583cc325383ddd126: Generated new TSK 0
I20260929 02:49:20.154421 7375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:49:20.205781 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38207
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:20.534894 7390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:20.535192 7390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:20.535295 7390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:20.542280 7390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:20.542459 7390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:20.542601 7390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:20.552042 7390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:20.553217 7390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:20.554436 7390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:20.567140 7395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:20.567200 7398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:20.567183 7396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:20.568248 7390 server_base.cc:1034] running on GCE node
I20260929 02:49:20.568863 7390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:20.569782 7390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:20.571007 7390 hybrid_clock.cc:648] HybridClock initialized: now 1790650160570953 us; error 39 us; skew 500 ppm
I20260929 02:49:20.571324 7390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:20 dist-test-slave-t6c1 krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650160, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:20.574579 7390 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:49:20.576181 7390 webserver.cc:460] Webserver started at http://127.4.197.129:35067/ using document root <none> and password file <none>
I20260929 02:49:20.576855 7390 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:20.576967 7390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:20.577244 7390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:20.580109 7390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "48af4256b42c4f48a48f273602c23efd"
format_stamp: "Formatted at 2026-09-29 02:49:20 on dist-test-slave-t6c1"
server_key: "30fd25e535b734fb78ec46ddf2669a61"
server_key_iv: "166cafa05944d02270d2a58b01dadf8c"
server_key_version: "encryptionkey@0"
I20260929 02:49:20.580839 7390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "48af4256b42c4f48a48f273602c23efd"
format_stamp: "Formatted at 2026-09-29 02:49:20 on dist-test-slave-t6c1"
server_key: "30fd25e535b734fb78ec46ddf2669a61"
server_key_iv: "166cafa05944d02270d2a58b01dadf8c"
server_key_version: "encryptionkey@0"
I20260929 02:49:20.585708 7390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:49:20.588959 7405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:20.590189 7390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:20.590330 7390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "48af4256b42c4f48a48f273602c23efd"
format_stamp: "Formatted at 2026-09-29 02:49:20 on dist-test-slave-t6c1"
server_key: "30fd25e535b734fb78ec46ddf2669a61"
server_key_iv: "166cafa05944d02270d2a58b01dadf8c"
server_key_version: "encryptionkey@0"
I20260929 02:49:20.590497 7390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:20.608856 7390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:20.612975 7390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:20.613230 7390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:20.614146 7390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:20.615533 7390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:20.615609 7390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:20.615732 7390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:20.615794 7390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:20.645488 7390 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:45295
I20260929 02:49:20.645545 7517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:45295 every 8 connection(s)
I20260929 02:49:20.646719 7390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:20.654035 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 7390
I20260929 02:49:20.654230 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:20.655022 4886 external_mini_cluster.cc:1428] Setting key 1ad70fcf1f9d1ed152c66cf7d84cb04b
Sep 29 02:49:20 dist-test-slave-t6c1 krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650160, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:20.666821 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:20.649193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:50239 (local address 127.4.197.190:38207)
0929 02:49:20.649607 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:20.649612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:20.650587 (+ 975us) server_negotiation.cc:407] Connection header received
0929 02:49:20.652425 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:20.652430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:20.652507 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:20.652629 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:20.654238 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:20.655306 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:20.657184 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:20.657769 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:20.657894 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:20.661212 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:20.661219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:20.661224 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:20.661267 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:20.663694 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:20.664575 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:20.664579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:20.664581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:20.664700 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:20.665150 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:20.665153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:20.665155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:20.665416 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:20.665523 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:20.666386 (+ 863us) server_negotiation.cc:299] Negotiation successful
0929 02:49:20.666559 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
I20260929 02:49:20.668344 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:20.649724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38207 (local address 127.4.197.129:50239)
0929 02:49:20.650421 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:20.650460 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:20.652128 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:20.652866 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:20.652874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:20.653405 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:20.654046 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:20.654066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:20.655456 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:20.655460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:20.657051 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:20.657064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:20.657980 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:20.657985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:20.658181 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:20.658857 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:20.658882 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:20.660976 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:20.663843 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:20.663852 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:20.663872 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:20.664453 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:20.664801 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:20.664805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:20.664808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:20.665052 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:20.665522 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:20.665530 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:20.665981 (+ 451us) client_negotiation.cc:769] Sending connection context
0929 02:49:20.666309 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 02:49:20.666864 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":255,"threads_started":1}
I20260929 02:49:20.670048 7518 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38207
I20260929 02:49:20.670406 7518 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:20.671375 7518 heartbeater.cc:511] Master 127.4.197.190:38207 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:49:20.673673 7329 ts_manager.cc:194] Registered new tserver with Master: 48af4256b42c4f48a48f273602c23efd (127.4.197.129:45295)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
I20260929 02:49:20.675287 7329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:50239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:49:20.694267 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38207
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:21.009969 7524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:21.010275 7524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:21.010416 7524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:21.017758 7524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:21.017884 7524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:21.018043 7524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:21.027901 7524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:21.029204 7524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:21.030531 7524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:21.043210 7530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:21.043237 7529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:21.043781 7532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:21.043871 7524 server_base.cc:1034] running on GCE node
I20260929 02:49:21.044518 7524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:21.045433 7524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:21.046655 7524 hybrid_clock.cc:648] HybridClock initialized: now 1790650161046599 us; error 49 us; skew 500 ppm
I20260929 02:49:21.047049 7524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:21 dist-test-slave-t6c1 krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:21.053426 7524 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:49:21.055133 7524 webserver.cc:460] Webserver started at http://127.4.197.130:46189/ using document root <none> and password file <none>
I20260929 02:49:21.055837 7524 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:21.055949 7524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:21.056228 7524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:21.059348 7524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "af356e0381eb4f16967582e73fe5a6df"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-t6c1"
server_key: "b42b378b9503194d97e5a410b343f6f1"
server_key_iv: "b42e22ef3cd3799ea7b1bebe24ed9296"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.060240 7524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "af356e0381eb4f16967582e73fe5a6df"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-t6c1"
server_key: "b42b378b9503194d97e5a410b343f6f1"
server_key_iv: "b42e22ef3cd3799ea7b1bebe24ed9296"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.065835 7524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:49:21.069551 7539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:21.071261 7524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:21.071434 7524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "af356e0381eb4f16967582e73fe5a6df"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-t6c1"
server_key: "b42b378b9503194d97e5a410b343f6f1"
server_key_iv: "b42e22ef3cd3799ea7b1bebe24ed9296"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.071646 7524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:21.083163 7524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:21.087330 7524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:21.087577 7524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:21.088485 7524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:21.089759 7524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:21.089833 7524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:21.089946 7524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:21.089996 7524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:21.119067 7524 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:43989
I20260929 02:49:21.119112 7651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:43989 every 8 connection(s)
I20260929 02:49:21.120262 7524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:21.130617 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 7524
I20260929 02:49:21.130837 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:21.131349 4886 external_mini_cluster.cc:1428] Setting key 9e011da1bf293367bdcf8e3a9969dcdb
Sep 29 02:49:21 dist-test-slave-t6c1 krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:21.140668 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:21.122602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:54501 (local address 127.4.197.190:38207)
0929 02:49:21.122808 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:21.122814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:21.124133 (+ 1319us) server_negotiation.cc:407] Connection header received
0929 02:49:21.125958 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:21.125963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:21.126034 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:21.126122 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:21.127684 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.128422 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.130167 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.130775 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.130914 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:21.134497 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:21.134504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:21.134508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:21.134554 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:21.137678 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:21.138430 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:21.138436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:21.138439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:21.138592 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:21.139066 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:21.139070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:21.139072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:21.139329 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:21.139440 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:21.140252 (+ 812us) server_negotiation.cc:299] Negotiation successful
0929 02:49:21.140425 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:49:21.142084 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:21.123241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38207 (local address 127.4.197.130:54501)
0929 02:49:21.123971 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:21.124020 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:21.125672 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:21.126342 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:21.126352 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:21.126872 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:21.127538 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:21.127557 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.128541 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.128545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:21.130035 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:21.130051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.131030 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.131037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:21.131245 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:21.131896 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:21.131919 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:21.134253 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:21.137845 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:21.137852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:21.137865 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:21.138307 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:21.138684 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:21.138687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:21.138689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:21.138962 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:21.139439 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:21.139448 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:21.139894 (+ 446us) client_negotiation.cc:769] Sending connection context
0929 02:49:21.140183 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 02:49:21.140720 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":277,"threads_started":1}
I20260929 02:49:21.143455 7652 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38207
I20260929 02:49:21.143786 7652 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:21.144487 7652 heartbeater.cc:511] Master 127.4.197.190:38207 requested a full tablet report, sending...
I20260929 02:49:21.145818 7329 ts_manager.cc:194] Registered new tserver with Master: af356e0381eb4f16967582e73fe5a6df (127.4.197.130:43989)
I20260929 02:49:21.146617 7329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:54501
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:49:21.170679 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38207
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:21.483688 7657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:21.483994 7657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:21.484133 7657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:21.491366 7657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:21.491498 7657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:21.491667 7657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:21.501657 7657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:21.503054 7657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:21.504382 7657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:21.516245 7662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:21.516386 7665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:21.516420 7663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:21.517362 7657 server_base.cc:1034] running on GCE node
I20260929 02:49:21.517910 7657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:21.518740 7657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:21.519918 7657 hybrid_clock.cc:648] HybridClock initialized: now 1790650161519865 us; error 46 us; skew 500 ppm
I20260929 02:49:21.520180 7657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:21 dist-test-slave-t6c1 krb5kdc[7284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:21.523041 7657 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:49:21.524384 7657 webserver.cc:460] Webserver started at http://127.4.197.131:45645/ using document root <none> and password file <none>
I20260929 02:49:21.525014 7657 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:21.525122 7657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:21.525393 7657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:21.528206 7657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "31d07339072442eb9ee46f6be633c576"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-t6c1"
server_key: "3722ecac43f2b734059b405b901d7f90"
server_key_iv: "33fe0e8337dc6aebe1eb6f262fcc343f"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.529111 7657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "31d07339072442eb9ee46f6be633c576"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-t6c1"
server_key: "3722ecac43f2b734059b405b901d7f90"
server_key_iv: "33fe0e8337dc6aebe1eb6f262fcc343f"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.534106 7657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:49:21.537500 7672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:21.538841 7657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:49:21.539036 7657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "31d07339072442eb9ee46f6be633c576"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-t6c1"
server_key: "3722ecac43f2b734059b405b901d7f90"
server_key_iv: "33fe0e8337dc6aebe1eb6f262fcc343f"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.539214 7657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:21.551015 7657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:21.554662 7657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:21.554987 7657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:21.555909 7657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:21.557185 7657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:21.557261 7657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:21.557386 7657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:21.557440 7657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:21.587363 7657 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:44945
I20260929 02:49:21.587402 7784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:44945 every 8 connection(s)
I20260929 02:49:21.588600 7657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:21.597124 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 7657
I20260929 02:49:21.597303 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:21.597839 4886 external_mini_cluster.cc:1428] Setting key 1d08c68669d89d1e2fb16a71ba3755ba
Sep 29 02:49:21 dist-test-slave-t6c1 krb5kdc[7284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:21.607764 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:21.591195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:48777 (local address 127.4.197.190:38207)
0929 02:49:21.591393 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:21.591398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:21.592688 (+ 1290us) server_negotiation.cc:407] Connection header received
0929 02:49:21.594519 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:21.594525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:21.594606 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:21.594689 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:21.596185 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.597061 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.598892 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.599378 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.599483 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:21.602621 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:21.602626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:21.602630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:21.602661 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:21.605027 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:21.605671 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:21.605676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:21.605679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:21.605790 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:21.606165 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:21.606170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:21.606173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:21.606427 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:21.606567 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:21.607222 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 02:49:21.607438 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:49:21.608649 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:21.591735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38207 (local address 127.4.197.131:48777)
0929 02:49:21.592499 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:21.592539 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:21.594255 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:21.594949 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:21.594960 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:21.595484 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:21.596026 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:21.596043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.597189 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.597193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:21.598767 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:21.598785 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.599580 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.599586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:21.599811 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:21.600422 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:21.600443 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:21.602423 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:21.605151 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:21.605160 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:21.605175 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:21.605568 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:21.605884 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:21.605889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:21.605893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:21.606066 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:21.606547 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:21.606554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:21.606850 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 02:49:21.607120 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 02:49:21.607612 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":299,"threads_started":1}
I20260929 02:49:21.609882 7785 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38207
I20260929 02:49:21.610136 7785 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:21.610733 7785 heartbeater.cc:511] Master 127.4.197.190:38207 requested a full tablet report, sending...
I20260929 02:49:21.612171 7329 ts_manager.cc:194] Registered new tserver with Master: 31d07339072442eb9ee46f6be633c576 (127.4.197.131:44945)
I20260929 02:49:21.612887 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:21.613122 7329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:48777
W20260929 02:49:21.640738 7520 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:49:21.638397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.4.197.190:38207)
0929 02:49:21.638628 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:21.638635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:21.638950 (+ 315us) server_negotiation.cc:407] Connection header received
0929 02:49:21.639146 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:21.639151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:21.639215 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:21.639288 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:21.640490 (+ 1202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38750: BlockingRecv error: recv got EOF from 127.0.0.1:38750 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:49:21.644435 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 7390
W20260929 02:49:21.650272 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 7390: Not authorized: Client connection negotiation failed: client connection to 127.4.197.129:45295: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:49:21.650463 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 7390
I20260929 02:49:21.662493 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 7524
W20260929 02:49:21.664909 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 7524: Not authorized: Client connection negotiation failed: client connection to 127.4.197.130:43989: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:49:21.665041 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 7524
I20260929 02:49:21.676623 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 7657
W20260929 02:49:21.679136 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 7657: Not authorized: Client connection negotiation failed: client connection to 127.4.197.131:44945: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:49:21.679270 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 7657
I20260929 02:49:21.690855 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 7297
I20260929 02:49:21.692085 7359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:21.795946 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 7297
2026-09-29T02:49:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3347 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:21 dist-test-slave-t6c1 krb5kdc[7802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:21 dist-test-slave-t6c1 krb5kdc[7802](info): set up 2 sockets
Sep 29 02:49:21 dist-test-slave-t6c1 krb5kdc[7802](info): commencing operation
krb5kdc: starting...
W20260929 02:49:22.783259 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:49:22 dist-test-slave-t6c1 krb5kdc[7802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:49:22.952169 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:37699
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:37699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:23.263212 7815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:23.263538 7815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:23.263675 7815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:23.271292 7815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:23.271422 7815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:23.271495 7815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:23.271572 7815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:23.271636 7815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:23.281908 7815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:37699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:37699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:23.283262 7815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:23.284579 7815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:23.295542 7821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:23.295644 7823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:23.295699 7820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:23.296749 7815 server_base.cc:1034] running on GCE node
I20260929 02:49:23.297564 7815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:23.298960 7815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:23.300171 7815 hybrid_clock.cc:648] HybridClock initialized: now 1790650163300124 us; error 44 us; skew 500 ppm
I20260929 02:49:23.300515 7815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:23 dist-test-slave-t6c1 krb5kdc[7802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650163, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:23.303560 7815 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:49:23.305007 7815 webserver.cc:460] Webserver started at http://127.4.197.190:42599/ using document root <none> and password file <none>
I20260929 02:49:23.305636 7815 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:23.305744 7815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:23.306026 7815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:23.308902 7815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "25162ba2e613426c86b87fca6509827a"
format_stamp: "Formatted at 2026-09-29 02:49:23 on dist-test-slave-t6c1"
server_key: "4dd912c6338d6b9fe8b26666bfffc10e"
server_key_iv: "7a9e848d19246de6db3d3b63c97336e5"
server_key_version: "encryptionkey@0"
I20260929 02:49:23.309643 7815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "25162ba2e613426c86b87fca6509827a"
format_stamp: "Formatted at 2026-09-29 02:49:23 on dist-test-slave-t6c1"
server_key: "4dd912c6338d6b9fe8b26666bfffc10e"
server_key_iv: "7a9e848d19246de6db3d3b63c97336e5"
server_key_version: "encryptionkey@0"
I20260929 02:49:23.314548 7815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:49:23.317766 7830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:23.319005 7815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:23.319139 7815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "25162ba2e613426c86b87fca6509827a"
format_stamp: "Formatted at 2026-09-29 02:49:23 on dist-test-slave-t6c1"
server_key: "4dd912c6338d6b9fe8b26666bfffc10e"
server_key_iv: "7a9e848d19246de6db3d3b63c97336e5"
server_key_version: "encryptionkey@0"
I20260929 02:49:23.319307 7815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:23.333568 7815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:23.337891 7815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:23.338155 7815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:23.355167 7815 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:37699
I20260929 02:49:23.355186 7881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:37699 every 8 connection(s)
I20260929 02:49:23.356540 7815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:23.358778 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 7815
I20260929 02:49:23.359004 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:23.359782 4886 external_mini_cluster.cc:1428] Setting key 67f338ec19a741b5c2984c4c95d5eb24
I20260929 02:49:23.360822 7882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:49:23 dist-test-slave-t6c1 krb5kdc[7802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:23.374382 7882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a: Bootstrap starting.
I20260929 02:49:23.378232 7882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:23.379405 7882 log.cc:824] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:23.382442 7882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a: No bootstrap required, opened a new log
I20260929 02:49:23.382601 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.361626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.4.197.190:37699)
0929 02:49:23.362362 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.362374 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.362413 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:49:23.364026 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.364050 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.364476 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.364837 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:23.366644 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.368091 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.369753 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.370435 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.370573 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.374875 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:23.374885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:23.374903 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:23.374970 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:23.378443 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:23.379094 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:23.379101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:23.379109 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:23.379309 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:23.379604 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:23.379609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:23.379612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:23.380256 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:23.380483 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.380952 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.381335 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":330,"threads_started":1}
I20260929 02:49:23.386228 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:49:23.389906 7882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25162ba2e613426c86b87fca6509827a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 37699 } }
I20260929 02:49:23.390213 7882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:23.390301 7882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25162ba2e613426c86b87fca6509827a, State: Initialized, Role: FOLLOWER
I20260929 02:49:23.390816 7882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25162ba2e613426c86b87fca6509827a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 37699 } }
I20260929 02:49:23.391070 7882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:23.391186 7882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:23.391345 7882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:23.393589 7882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25162ba2e613426c86b87fca6509827a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 37699 } }
I20260929 02:49:23.394019 7882 leader_election.cc:302] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25162ba2e613426c86b87fca6509827a; no voters:
I20260929 02:49:23.394562 7882 leader_election.cc:288] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:23.394649 7888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:23.395588 7888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [term 1 LEADER]: Becoming Leader. State: Replica: 25162ba2e613426c86b87fca6509827a, State: Running, Role: LEADER
I20260929 02:49:23.396132 7888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25162ba2e613426c86b87fca6509827a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 37699 } }
I20260929 02:49:23.396615 7882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:23.398622 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25162ba2e613426c86b87fca6509827a. Latest consensus state: current_term: 1 leader_uuid: "25162ba2e613426c86b87fca6509827a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25162ba2e613426c86b87fca6509827a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 37699 } } }
I20260929 02:49:23.398739 7889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25162ba2e613426c86b87fca6509827a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25162ba2e613426c86b87fca6509827a" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 37699 } } }
I20260929 02:49:23.399004 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:23.399081 7889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:23.403911 7896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:23.409039 7896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:23.422235 7896 catalog_manager.cc:1347] Generated new cluster ID: 1d4eb4cc03054b81a06e733c0593d816
I20260929 02:49:23.422336 7896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:23.425231 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.416403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60950 (local address 127.4.197.190:37699)
0929 02:49:23.416586 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.416593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.417673 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 02:49:23.417855 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.417861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.417945 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.418055 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:23.419249 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.420424 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.422149 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.422739 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.422861 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.423968 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:23.423978 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:23.423982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:23.424514 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:23.424554 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:23.424580 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:23.424674 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.424795 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.424885 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:49:23.433446 7896 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:23.435200 7896 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:23.446386 7896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25162ba2e613426c86b87fca6509827a: Generated new TSK 0
I20260929 02:49:23.447425 7896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:49:23.460350 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 7815
I20260929 02:49:23.461401 7877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:23.548432 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 7815
2026-09-29T02:49:23Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1747 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:23 dist-test-slave-t6c1 krb5kdc[7916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:23 dist-test-slave-t6c1 krb5kdc[7916](info): set up 2 sockets
Sep 29 02:49:23 dist-test-slave-t6c1 krb5kdc[7916](info): commencing operation
krb5kdc: starting...
W20260929 02:49:24.527027 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/test-user.keytab.
W20260929 02:49:25.287650 4886 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.684s user 0.000s sys 0.004s
Sep 29 02:49:25 dist-test-slave-t6c1 krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
W20260929 02:49:26.401604 4886 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.197.190: real 1.069s user 0.000s sys 0.008s
I20260929 02:49:26.402604 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:40163
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:40163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:26.713776 7929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:26.714071 7929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:26.714196 7929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:26.721204 7929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:26.721299 7929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:26.721354 7929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:26.721431 7929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:26.721481 7929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:26.731477 7929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:40163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:40163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:26.732860 7929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:26.734189 7929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:26.745887 7937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:26.745981 7934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:26.745959 7935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:26.747084 7929 server_base.cc:1034] running on GCE node
I20260929 02:49:26.747862 7929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:26.749281 7929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:26.750540 7929 hybrid_clock.cc:648] HybridClock initialized: now 1790650166750480 us; error 41 us; skew 500 ppm
I20260929 02:49:26.750900 7929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:26 dist-test-slave-t6c1 krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650166, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:26.757251 7929 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:49:26.758687 7929 webserver.cc:460] Webserver started at http://127.4.197.190:36461/ using document root <none> and password file <none>
I20260929 02:49:26.759374 7929 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:26.759475 7929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:26.759716 7929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:26.762609 7929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "278614e2a05e4b308d7f6eee193e66a0"
format_stamp: "Formatted at 2026-09-29 02:49:26 on dist-test-slave-t6c1"
server_key: "4055a4e5fab110b191747e9427396f1c"
server_key_iv: "d828841f113c17d2a0b93579facaf83f"
server_key_version: "encryptionkey@0"
I20260929 02:49:26.763336 7929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "278614e2a05e4b308d7f6eee193e66a0"
format_stamp: "Formatted at 2026-09-29 02:49:26 on dist-test-slave-t6c1"
server_key: "4055a4e5fab110b191747e9427396f1c"
server_key_iv: "d828841f113c17d2a0b93579facaf83f"
server_key_version: "encryptionkey@0"
I20260929 02:49:26.768224 7929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:49:26.771569 7944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:26.772895 7929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:26.773057 7929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "278614e2a05e4b308d7f6eee193e66a0"
format_stamp: "Formatted at 2026-09-29 02:49:26 on dist-test-slave-t6c1"
server_key: "4055a4e5fab110b191747e9427396f1c"
server_key_iv: "d828841f113c17d2a0b93579facaf83f"
server_key_version: "encryptionkey@0"
I20260929 02:49:26.773227 7929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:26.794137 7929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:26.798487 7929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:26.798784 7929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:26.817284 7929 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:40163
I20260929 02:49:26.817305 7995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:40163 every 8 connection(s)
I20260929 02:49:26.818629 7929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:26.819346 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 7929
I20260929 02:49:26.819540 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:26.820245 4886 external_mini_cluster.cc:1428] Setting key 6a7f8ecfd09b3a9bbb5e54be0d134536
I20260929 02:49:26.823275 7996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:49:26 dist-test-slave-t6c1 krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:26.834388 7996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0: Bootstrap starting.
I20260929 02:49:26.837582 7996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:26.838613 7996 log.cc:824] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:26.841572 7996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0: No bootstrap required, opened a new log
I20260929 02:49:26.842128 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.822383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.4.197.190:40163)
0929 02:49:26.823215 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.823228 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.823276 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:49:26.824936 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.824961 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.825441 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.825869 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:26.827372 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.828747 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.830059 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.830751 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.830940 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.835117 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:26.835126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:26.835142 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:26.835179 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:26.838088 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.838759 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.838767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.838776 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.838996 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.839374 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.839379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.839383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.839955 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:26.840233 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.840673 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.841095 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":626,"thread_start_us":355,"threads_started":1}
I20260929 02:49:26.849970 7996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278614e2a05e4b308d7f6eee193e66a0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40163 } }
I20260929 02:49:26.850330 7996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:26.850421 7996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278614e2a05e4b308d7f6eee193e66a0, State: Initialized, Role: FOLLOWER
I20260929 02:49:26.850987 7996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278614e2a05e4b308d7f6eee193e66a0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40163 } }
I20260929 02:49:26.851253 7996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:26.851387 7996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:26.851564 7996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:26.853662 7996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278614e2a05e4b308d7f6eee193e66a0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40163 } }
I20260929 02:49:26.854113 7996 leader_election.cc:302] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 278614e2a05e4b308d7f6eee193e66a0; no voters:
I20260929 02:49:26.854666 7996 leader_election.cc:288] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:26.854790 8001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:26.855417 8001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [term 1 LEADER]: Becoming Leader. State: Replica: 278614e2a05e4b308d7f6eee193e66a0, State: Running, Role: LEADER
I20260929 02:49:26.855924 8001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278614e2a05e4b308d7f6eee193e66a0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40163 } }
I20260929 02:49:26.856660 7996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:26.858711 8002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "278614e2a05e4b308d7f6eee193e66a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278614e2a05e4b308d7f6eee193e66a0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40163 } } }
I20260929 02:49:26.858745 8003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 278614e2a05e4b308d7f6eee193e66a0. Latest consensus state: current_term: 1 leader_uuid: "278614e2a05e4b308d7f6eee193e66a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278614e2a05e4b308d7f6eee193e66a0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40163 } } }
I20260929 02:49:26.859122 8003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:26.859122 8002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:26.860900 8007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:26.866231 8007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:26.876420 8007 catalog_manager.cc:1347] Generated new cluster ID: 840d28d01228425998c4f29378e756e1
I20260929 02:49:26.876547 8007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:26.903993 8007 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:26.905300 8007 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:26.912528 8007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 278614e2a05e4b308d7f6eee193e66a0: Generated new TSK 0
I20260929 02:49:26.913692 8007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:49:26.966722 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40163
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:27.277972 8022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:27.278251 8022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:27.278373 8022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:27.285595 8022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:27.285702 8022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:27.285854 8022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:27.295437 8022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:27.296679 8022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:27.297981 8022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:27.310079 8028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:27.310199 8030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:27.310185 8027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:27.311160 8022 server_base.cc:1034] running on GCE node
I20260929 02:49:27.311669 8022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:27.312494 8022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:27.313687 8022 hybrid_clock.cc:648] HybridClock initialized: now 1790650167313633 us; error 46 us; skew 500 ppm
I20260929 02:49:27.313953 8022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:27 dist-test-slave-t6c1 krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650167, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:27.316879 8022 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:49:27.318099 8022 webserver.cc:460] Webserver started at http://127.4.197.129:39297/ using document root <none> and password file <none>
I20260929 02:49:27.318697 8022 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:27.318792 8022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:27.319067 8022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:27.321719 8022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "c9f9a650da074ef994490b7d82ba0576"
format_stamp: "Formatted at 2026-09-29 02:49:27 on dist-test-slave-t6c1"
server_key: "e305f30efe362a7d5473342fe2744f50"
server_key_iv: "dd41bda70a3c4470ad11fc170e14fb52"
server_key_version: "encryptionkey@0"
I20260929 02:49:27.322427 8022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "c9f9a650da074ef994490b7d82ba0576"
format_stamp: "Formatted at 2026-09-29 02:49:27 on dist-test-slave-t6c1"
server_key: "e305f30efe362a7d5473342fe2744f50"
server_key_iv: "dd41bda70a3c4470ad11fc170e14fb52"
server_key_version: "encryptionkey@0"
I20260929 02:49:27.328388 8022 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 02:49:27.333297 8037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:27.334729 8022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:49:27.334879 8022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "c9f9a650da074ef994490b7d82ba0576"
format_stamp: "Formatted at 2026-09-29 02:49:27 on dist-test-slave-t6c1"
server_key: "e305f30efe362a7d5473342fe2744f50"
server_key_iv: "dd41bda70a3c4470ad11fc170e14fb52"
server_key_version: "encryptionkey@0"
I20260929 02:49:27.335116 8022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:27.347543 8022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:27.351577 8022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:27.351856 8022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:27.352800 8022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:27.354130 8022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:27.354200 8022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:27.354323 8022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:27.354377 8022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:27.383737 8022 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:41651
I20260929 02:49:27.383777 8149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:41651 every 8 connection(s)
I20260929 02:49:27.384979 8022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:27.395141 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 8022
I20260929 02:49:27.395326 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:27.395931 4886 external_mini_cluster.cc:1428] Setting key c92fd924d41c00577e591e05c85e657a
Sep 29 02:49:27 dist-test-slave-t6c1 krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650167, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:27.403939 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:27.387377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:48495 (local address 127.4.197.190:40163)
0929 02:49:27.387813 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:27.387820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:27.388834 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 02:49:27.390602 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:27.390606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:27.390677 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:27.390774 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:27.392373 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:27.393096 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:27.394600 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:27.395074 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:27.395186 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:27.398653 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:27.398658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:27.398662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:27.398693 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:27.401139 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:27.401789 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:27.401794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:27.401796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:27.401919 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:27.402297 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:27.402302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:27.402306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:27.402581 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:27.402709 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:27.403383 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 02:49:27.403593 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":196,"threads_started":1}
I20260929 02:49:27.404907 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:27.387977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:40163 (local address 127.4.197.129:48495)
0929 02:49:27.388664 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:27.388705 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:27.390344 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:27.391039 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:27.391049 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:27.391577 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:27.392225 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:27.392244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:27.393223 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:27.393226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:27.394465 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:27.394477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:27.395292 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:27.395299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:27.395562 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:27.396201 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:27.396224 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:27.398445 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:27.401263 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:27.401272 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:27.401289 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:27.401685 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:27.402015 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:27.402020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:27.402024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:27.402195 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:27.402697 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:27.402705 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:27.403056 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 02:49:27.403290 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 02:49:27.403796 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":260,"threads_started":1}
I20260929 02:49:27.406340 8150 heartbeater.cc:347] Connected to a master server at 127.4.197.190:40163
I20260929 02:49:27.406605 8150 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:27.407289 8150 heartbeater.cc:511] Master 127.4.197.190:40163 requested a full tablet report, sending...
I20260929 02:49:27.409461 7961 ts_manager.cc:194] Registered new tserver with Master: c9f9a650da074ef994490b7d82ba0576 (127.4.197.129:41651)
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:49:27.411476 7961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:48495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:49:27.433689 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40163
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:27.754997 8156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:27.755295 8156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:27.755427 8156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:27.762995 8156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:27.763123 8156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:27.763283 8156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:27.773389 8156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:27.774865 8156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:27.776429 8156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:27.789458 8161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:27.789492 8164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:27.789428 8162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:27.790186 8156 server_base.cc:1034] running on GCE node
I20260929 02:49:27.790864 8156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:27.791838 8156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:27.793043 8156 hybrid_clock.cc:648] HybridClock initialized: now 1790650167792995 us; error 44 us; skew 500 ppm
I20260929 02:49:27.793335 8156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:27 dist-test-slave-t6c1 krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650167, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:27.796341 8156 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:49:27.797700 8156 webserver.cc:460] Webserver started at http://127.4.197.130:39987/ using document root <none> and password file <none>
I20260929 02:49:27.798357 8156 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:27.798468 8156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:27.798736 8156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:27.801571 8156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "8c22acc134a34e938a0afd39f98d47dd"
format_stamp: "Formatted at 2026-09-29 02:49:27 on dist-test-slave-t6c1"
server_key: "2ce0f361c8efa909ba17f31aaaaa72fb"
server_key_iv: "08b08905522b17215c0af4d09d254f91"
server_key_version: "encryptionkey@0"
I20260929 02:49:27.802278 8156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "8c22acc134a34e938a0afd39f98d47dd"
format_stamp: "Formatted at 2026-09-29 02:49:27 on dist-test-slave-t6c1"
server_key: "2ce0f361c8efa909ba17f31aaaaa72fb"
server_key_iv: "08b08905522b17215c0af4d09d254f91"
server_key_version: "encryptionkey@0"
I20260929 02:49:27.807104 8156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260929 02:49:27.810281 8171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:27.811529 8156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:27.811666 8156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "8c22acc134a34e938a0afd39f98d47dd"
format_stamp: "Formatted at 2026-09-29 02:49:27 on dist-test-slave-t6c1"
server_key: "2ce0f361c8efa909ba17f31aaaaa72fb"
server_key_iv: "08b08905522b17215c0af4d09d254f91"
server_key_version: "encryptionkey@0"
I20260929 02:49:27.811818 8156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:27.843793 8156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:27.848078 8156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:27.848366 8156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:27.849341 8156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:27.850726 8156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:27.850803 8156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:27.850947 8156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:27.851011 8156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 02:49:27.879151 8156 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:34949
I20260929 02:49:27.879196 8283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:34949 every 8 connection(s)
I20260929 02:49:27.880364 8156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:27.881559 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 8156
I20260929 02:49:27.881738 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:27.882266 4886 external_mini_cluster.cc:1428] Setting key 06cad94be2c58323903dd930808058d1
Sep 29 02:49:27 dist-test-slave-t6c1 krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650167, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
I20260929 02:49:27.901896 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:27.882998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:55241 (local address 127.4.197.190:40163)
0929 02:49:27.883183 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:27.883188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:27.884522 (+ 1334us) server_negotiation.cc:407] Connection header received
0929 02:49:27.887040 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:27.887044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:27.887111 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:27.887193 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:27.889192 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:27.889937 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:27.891575 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:27.892224 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:27.892347 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:27.895689 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:27.895697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:27.895702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:27.895751 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:27.898840 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:27.899578 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:27.899584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:27.899587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:27.899756 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:27.900137 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:27.900143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:27.900146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:27.900471 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:27.900623 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:27.901306 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 02:49:27.901638 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:49:27.902899 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:27.883604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:40163 (local address 127.4.197.130:55241)
0929 02:49:27.884355 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:27.884411 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:27.886692 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:27.887462 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:27.887475 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:27.888174 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:27.888988 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:27.889014 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:27.890082 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:27.890088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:27.891429 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:27.891445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:27.892409 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:27.892414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:27.892608 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:27.893223 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:27.893245 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:27.895479 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:27.899019 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:27.899026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:27.899040 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:27.899450 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:27.899852 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:27.899855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:27.899858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:27.900023 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:27.900600 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:27.900608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:27.900937 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 02:49:27.901158 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 02:49:27.901624 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":288,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
I20260929 02:49:27.904449 8284 heartbeater.cc:347] Connected to a master server at 127.4.197.190:40163
I20260929 02:49:27.904775 8284 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:27.905530 8284 heartbeater.cc:511] Master 127.4.197.190:40163 requested a full tablet report, sending...
I20260929 02:49:27.907096 7961 ts_manager.cc:194] Registered new tserver with Master: 8c22acc134a34e938a0afd39f98d47dd (127.4.197.130:34949)
I20260929 02:49:27.907936 7961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:55241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:49:27.926092 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40163
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:28.237028 8289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:28.237344 8289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:28.237485 8289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:28.244766 8289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:28.244895 8289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:28.245052 8289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:28.254639 8289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:28.255923 8289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:28.257259 8289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:28.269800 8297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:28.269892 8295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:28.269829 8294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:28.270480 8289 server_base.cc:1034] running on GCE node
I20260929 02:49:28.271289 8289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:28.272320 8289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:28.273542 8289 hybrid_clock.cc:648] HybridClock initialized: now 1790650168273489 us; error 41 us; skew 500 ppm
I20260929 02:49:28.273907 8289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:28 dist-test-slave-t6c1 krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650168, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:28.277201 8289 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:49:28.278625 8289 webserver.cc:460] Webserver started at http://127.4.197.131:33323/ using document root <none> and password file <none>
I20260929 02:49:28.279348 8289 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:28.279474 8289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:28.279749 8289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:28.282546 8289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "8f8ead77b8bc45529f129651bf04ade1"
format_stamp: "Formatted at 2026-09-29 02:49:28 on dist-test-slave-t6c1"
server_key: "d8de84f132a4ec9898a1b2e2140f341f"
server_key_iv: "2d6e6d48ade87acba36b5342cd23e0d1"
server_key_version: "encryptionkey@0"
I20260929 02:49:28.283331 8289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "8f8ead77b8bc45529f129651bf04ade1"
format_stamp: "Formatted at 2026-09-29 02:49:28 on dist-test-slave-t6c1"
server_key: "d8de84f132a4ec9898a1b2e2140f341f"
server_key_iv: "2d6e6d48ade87acba36b5342cd23e0d1"
server_key_version: "encryptionkey@0"
I20260929 02:49:28.288271 8289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:49:28.291512 8304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:28.292796 8289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:28.292934 8289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "8f8ead77b8bc45529f129651bf04ade1"
format_stamp: "Formatted at 2026-09-29 02:49:28 on dist-test-slave-t6c1"
server_key: "d8de84f132a4ec9898a1b2e2140f341f"
server_key_iv: "2d6e6d48ade87acba36b5342cd23e0d1"
server_key_version: "encryptionkey@0"
I20260929 02:49:28.293105 8289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:28.320911 8289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:28.325250 8289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:28.325533 8289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:28.326494 8289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:28.327909 8289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:28.327986 8289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:28.328106 8289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:28.328159 8289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:28.357789 8289 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:42373
I20260929 02:49:28.357824 8416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:42373 every 8 connection(s)
I20260929 02:49:28.359115 8289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:28.362485 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 8289
I20260929 02:49:28.362680 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:28.363282 4886 external_mini_cluster.cc:1428] Setting key f2f4aedb188ec6b2b28b98c83e251e35
Sep 29 02:49:28 dist-test-slave-t6c1 krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650168, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:28.379731 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.361670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:33579 (local address 127.4.197.190:40163)
0929 02:49:28.361837 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:28.361842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:28.363338 (+ 1496us) server_negotiation.cc:407] Connection header received
0929 02:49:28.365395 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:28.365399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:28.365488 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:28.365575 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:28.367230 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.367964 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.369529 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.369940 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.370046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:28.373449 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:28.373454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:28.373458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:28.373491 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:28.376138 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:28.377094 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:28.377100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:28.377103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:28.377267 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:28.377882 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:28.377887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:28.377890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:28.378261 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:28.378427 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:28.379295 (+ 868us) server_negotiation.cc:299] Negotiation successful
0929 02:49:28.379475 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:49:28.380756 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.362254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:40163 (local address 127.4.197.131:33579)
0929 02:49:28.363167 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:28.363207 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:28.365096 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:28.365839 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:28.365848 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:28.366438 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:28.367078 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:28.367099 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.368109 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.368113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:28.369390 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:28.369402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.370157 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.370164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:28.370403 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:28.371101 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:28.371124 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:28.373223 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:28.376318 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:28.376329 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:28.376349 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:28.376958 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:28.377419 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:28.377424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:28.377449 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:28.377730 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:28.378396 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:28.378405 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:28.378895 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 02:49:28.379196 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 02:49:28.379640 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":331,"threads_started":1}
I20260929 02:49:28.382025 8417 heartbeater.cc:347] Connected to a master server at 127.4.197.190:40163
I20260929 02:49:28.382334 8417 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:28.383100 8417 heartbeater.cc:511] Master 127.4.197.190:40163 requested a full tablet report, sending...
I20260929 02:49:28.384781 7961 ts_manager.cc:194] Registered new tserver with Master: 8f8ead77b8bc45529f129651bf04ade1 (127.4.197.131:42373)
I20260929 02:49:28.385876 7961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:33579
I20260929 02:49:28.391381 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:49:28 dist-test-slave-t6c1 krb5kdc[7916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650168, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260929 02:49:28.414438 8150 heartbeater.cc:503] Master 127.4.197.190:40163 was elected leader, sending a full tablet report...
Sep 29 02:49:28 dist-test-slave-t6c1 krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650168, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:28.434463 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.421014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.4.197.190:40163)
0929 02:49:28.421183 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:28.421189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:28.421584 (+ 395us) server_negotiation.cc:407] Connection header received
0929 02:49:28.421743 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:28.421748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:28.421839 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:28.421923 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:28.423336 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.424058 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.425284 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.425688 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.425808 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:28.429479 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:28.429484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:28.429487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:28.429523 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:28.431830 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:28.432541 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:28.432546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:28.432548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:28.432685 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:28.433081 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:28.433085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:28.433087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:28.433405 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:28.433626 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:28.433934 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 02:49:28.434158 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 02:49:28.435038 7961 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54642
I20260929 02:49:28.436441 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8022
Sep 29 02:49:28 dist-test-slave-t6c1 krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650168, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:28.452584 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.437596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.4.197.129:41651)
0929 02:49:28.438270 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:28.438278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:28.438305 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:49:28.438366 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:28.438373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:28.438585 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:28.438731 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:28.439919 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.440969 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.442213 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.442897 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.443059 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:28.446876 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:28.446890 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:28.446905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:28.447003 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:28.449982 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:28.450685 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:28.450694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:28.450696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:28.450843 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:28.451206 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:28.451209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:28.451211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:28.451586 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:28.451802 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:28.452055 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 02:49:28.452278 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":324,"threads_started":1}
W20260929 02:49:28.453819 8145 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35270
W20260929 02:49:28.454897 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8022: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:49:28.455123 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 8022
I20260929 02:49:28.467500 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8156
Sep 29 02:49:28 dist-test-slave-t6c1 krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650168, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:49:28.484035 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.468506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.4.197.130:34949)
0929 02:49:28.469088 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:28.469097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:28.469123 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:49:28.469194 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:28.469202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:28.469505 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:28.469668 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:28.471196 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.472238 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.473383 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.473923 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.474020 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:28.478123 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:28.478138 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:28.478155 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:28.478201 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:28.480989 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:28.481702 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:28.481710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:28.481712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:28.481867 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:28.482289 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:28.482294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:28.482298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:28.482698 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:28.483028 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:28.483417 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 02:49:28.483628 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":274,"threads_started":1}
W20260929 02:49:28.485276 8279 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32876
W20260929 02:49:28.486428 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8156: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:49:28.486575 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 8156
I20260929 02:49:28.498315 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8289
Sep 29 02:49:28 dist-test-slave-t6c1 krb5kdc[7916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650168, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:49:28.514339 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.499278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.4.197.131:42373)
0929 02:49:28.499929 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:28.499936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:28.499967 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:49:28.500032 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:28.500038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:28.500260 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:28.500417 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:28.501846 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.502966 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.504264 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:28.504839 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:28.504960 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:28.509288 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:28.509301 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:28.509315 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:28.509351 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:28.511971 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:28.512582 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:28.512589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:28.512592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:28.512711 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:28.512998 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:28.513002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:28.513004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:28.513354 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:28.513589 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:28.513798 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 02:49:28.513987 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":341,"threads_started":1}
W20260929 02:49:28.515719 8412 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35324
W20260929 02:49:28.516748 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8289: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:49:28.516896 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 8289
I20260929 02:49:28.528476 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 7929
I20260929 02:49:28.529654 7991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:28.630050 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 7929
2026-09-29T02:49:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5087 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:28 dist-test-slave-t6c1 krb5kdc[8434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:28 dist-test-slave-t6c1 krb5kdc[8434](info): set up 2 sockets
Sep 29 02:49:28 dist-test-slave-t6c1 krb5kdc[8434](info): commencing operation
krb5kdc: starting...
W20260929 02:49:29.619122 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:49:29 dist-test-slave-t6c1 krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:49:29.804508 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36505
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:36505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:30.124538 8447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:30.124861 8447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:30.124950 8447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:30.132261 8447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:30.132401 8447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:30.132447 8447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:30.132505 8447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:30.132550 8447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:30.142979 8447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:36505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:30.144273 8447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:30.145509 8447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:30.156922 8452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:30.156883 8455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:30.156893 8453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:30.157599 8447 server_base.cc:1034] running on GCE node
I20260929 02:49:30.158383 8447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:30.159765 8447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:30.160954 8447 hybrid_clock.cc:648] HybridClock initialized: now 1790650170160908 us; error 30 us; skew 500 ppm
I20260929 02:49:30.161285 8447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:30 dist-test-slave-t6c1 krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650170, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:30.164184 8447 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:49:30.165534 8447 webserver.cc:460] Webserver started at http://127.4.197.190:41243/ using document root <none> and password file <none>
I20260929 02:49:30.166275 8447 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:30.166396 8447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:30.166700 8447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:30.169883 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "ef5e6c45948944c7b517a1c60864913e"
format_stamp: "Formatted at 2026-09-29 02:49:30 on dist-test-slave-t6c1"
server_key: "e1dd4c06cb98ce9bd20b5b5bc0d1c4d7"
server_key_iv: "fb8af4fd3309c5990b9ea2d5ea8733b4"
server_key_version: "encryptionkey@0"
I20260929 02:49:30.170771 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "ef5e6c45948944c7b517a1c60864913e"
format_stamp: "Formatted at 2026-09-29 02:49:30 on dist-test-slave-t6c1"
server_key: "e1dd4c06cb98ce9bd20b5b5bc0d1c4d7"
server_key_iv: "fb8af4fd3309c5990b9ea2d5ea8733b4"
server_key_version: "encryptionkey@0"
I20260929 02:49:30.176384 8447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 02:49:30.179961 8462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:30.181412 8447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:49:30.181597 8447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "ef5e6c45948944c7b517a1c60864913e"
format_stamp: "Formatted at 2026-09-29 02:49:30 on dist-test-slave-t6c1"
server_key: "e1dd4c06cb98ce9bd20b5b5bc0d1c4d7"
server_key_iv: "fb8af4fd3309c5990b9ea2d5ea8733b4"
server_key_version: "encryptionkey@0"
I20260929 02:49:30.181802 8447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:30.208767 8447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:30.212845 8447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:30.213117 8447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:30.231566 8447 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:36505
I20260929 02:49:30.231585 8513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:36505 every 8 connection(s)
I20260929 02:49:30.232993 8447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:30.236652 8514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:30.241824 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 8447
I20260929 02:49:30.242045 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:30.242640 4886 external_mini_cluster.cc:1428] Setting key cbf7662ce1b2e4b1f8217171eafbeefd
I20260929 02:49:30.246876 8514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e: Bootstrap starting.
I20260929 02:49:30.250219 8514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:30.251237 8514 log.cc:824] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:30.254012 8514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e: No bootstrap required, opened a new log
Sep 29 02:49:30 dist-test-slave-t6c1 krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:30.263170 8514 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5e6c45948944c7b517a1c60864913e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36505 } }
I20260929 02:49:30.263609 8514 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:30.263757 8514 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5e6c45948944c7b517a1c60864913e, State: Initialized, Role: FOLLOWER
I20260929 02:49:30.264483 8514 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5e6c45948944c7b517a1c60864913e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36505 } }
I20260929 02:49:30.264761 8514 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:30.264901 8514 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:30.265080 8514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:30.266326 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:30.244624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.4.197.190:36505)
0929 02:49:30.245413 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:30.245424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:30.245462 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:49:30.247129 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:30.247151 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:30.247531 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:30.247823 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:30.249565 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:30.251025 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:30.252504 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:30.253157 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:30.253294 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:30.257931 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:30.257939 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:30.257954 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:30.257991 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:30.261738 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:30.262406 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:30.262414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:30.262422 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:30.262630 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:30.263049 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:30.263054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:30.263057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:30.263763 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:30.264050 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:30.264441 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 02:49:30.264893 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":358,"threads_started":1}
I20260929 02:49:30.268252 8514 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5e6c45948944c7b517a1c60864913e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36505 } }
I20260929 02:49:30.268772 8514 leader_election.cc:302] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef5e6c45948944c7b517a1c60864913e; no voters:
I20260929 02:49:30.269410 8514 leader_election.cc:288] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:30.269562 8519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:30.270254 8519 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [term 1 LEADER]: Becoming Leader. State: Replica: ef5e6c45948944c7b517a1c60864913e, State: Running, Role: LEADER
I20260929 02:49:30.270780 8519 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5e6c45948944c7b517a1c60864913e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36505 } }
I20260929 02:49:30.271485 8514 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:30.273254 8520 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef5e6c45948944c7b517a1c60864913e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5e6c45948944c7b517a1c60864913e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36505 } } }
I20260929 02:49:30.273553 8520 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:30.274325 8525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:30.274184 8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef5e6c45948944c7b517a1c60864913e. Latest consensus state: current_term: 1 leader_uuid: "ef5e6c45948944c7b517a1c60864913e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5e6c45948944c7b517a1c60864913e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36505 } } }
I20260929 02:49:30.274478 8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:30.283402 8525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:30.293771 8525 catalog_manager.cc:1347] Generated new cluster ID: 6bf88ef6f5dd4dde8d006229a86a69d1
I20260929 02:49:30.293895 8525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:30.310712 8525 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:30.312168 8525 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:30.321349 8525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef5e6c45948944c7b517a1c60864913e: Generated new TSK 0
I20260929 02:49:30.322243 8525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:49:30.371156 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36505
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:30.682066 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:30.682327 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:30.682458 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:30.689494 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:30.689587 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:30.689744 8540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:30.699241 8540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:30.700507 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:30.701800 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:30.714588 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:30.714517 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:30.714617 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:30.715499 8540 server_base.cc:1034] running on GCE node
I20260929 02:49:30.716207 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:30.717262 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:30.718458 8540 hybrid_clock.cc:648] HybridClock initialized: now 1790650170718401 us; error 54 us; skew 500 ppm
I20260929 02:49:30.718791 8540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:30 dist-test-slave-t6c1 krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650170, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:30.721946 8540 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:49:30.723472 8540 webserver.cc:460] Webserver started at http://127.4.197.129:44595/ using document root <none> and password file <none>
I20260929 02:49:30.724141 8540 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:30.724239 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:30.724491 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:30.727473 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "55dd6371283741d4a2f5067a657c317b"
format_stamp: "Formatted at 2026-09-29 02:49:30 on dist-test-slave-t6c1"
server_key: "b141f8c0cdb53fd1514c3b7880053663"
server_key_iv: "f68f3092cf88b5de7d684216527272a6"
server_key_version: "encryptionkey@0"
I20260929 02:49:30.728211 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "55dd6371283741d4a2f5067a657c317b"
format_stamp: "Formatted at 2026-09-29 02:49:30 on dist-test-slave-t6c1"
server_key: "b141f8c0cdb53fd1514c3b7880053663"
server_key_iv: "f68f3092cf88b5de7d684216527272a6"
server_key_version: "encryptionkey@0"
I20260929 02:49:30.733278 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:49:30.736645 8555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:30.737957 8540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:49:30.738077 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "55dd6371283741d4a2f5067a657c317b"
format_stamp: "Formatted at 2026-09-29 02:49:30 on dist-test-slave-t6c1"
server_key: "b141f8c0cdb53fd1514c3b7880053663"
server_key_iv: "f68f3092cf88b5de7d684216527272a6"
server_key_version: "encryptionkey@0"
I20260929 02:49:30.738234 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:30.760385 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:30.765419 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:30.765676 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:30.766666 8540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:30.768209 8540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:30.768286 8540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:30.768378 8540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:30.768424 8540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:30.797577 8540 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:45317
I20260929 02:49:30.797616 8667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:45317 every 8 connection(s)
I20260929 02:49:30.798911 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:30.809111 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 8540
I20260929 02:49:30.809305 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:30.809911 4886 external_mini_cluster.cc:1428] Setting key 9b6bd2eae79f15fb7b661152aa2f1c49
Sep 29 02:49:30 dist-test-slave-t6c1 krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650170, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:30.817240 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:30.801287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:46877 (local address 127.4.197.190:36505)
0929 02:49:30.801697 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:30.801702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:30.802578 (+ 876us) server_negotiation.cc:407] Connection header received
0929 02:49:30.804389 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:30.804395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:30.804475 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:30.804567 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:30.806100 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:30.806840 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:30.808340 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:30.808795 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:30.808924 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:30.812194 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:30.812198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:30.812202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:30.812233 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:30.814479 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:30.815172 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:30.815175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:30.815178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:30.815287 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:30.815670 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:30.815673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:30.815675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:30.815929 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:30.816053 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:30.816686 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 02:49:30.816905 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":220,"threads_started":1}
I20260929 02:49:30.818121 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:30.801754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36505 (local address 127.4.197.129:46877)
0929 02:49:30.802415 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:30.802466 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:30.804120 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:30.804820 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:30.804829 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:30.805335 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:30.805925 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:30.805947 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:30.807006 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:30.807009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:30.808197 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:30.808209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:30.808995 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:30.809001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:30.809205 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:30.809825 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:30.809847 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:30.811990 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:30.814598 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:30.814606 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:30.814621 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:30.815063 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:30.815388 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:30.815393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:30.815397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:30.815563 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:30.816041 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:30.816048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:30.816347 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 02:49:30.816579 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 02:49:30.817054 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":248,"threads_started":1}
I20260929 02:49:30.819522 8668 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36505
I20260929 02:49:30.819804 8668 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:30.820546 8668 heartbeater.cc:511] Master 127.4.197.190:36505 requested a full tablet report, sending...
I20260929 02:49:30.822616 8479 ts_manager.cc:194] Registered new tserver with Master: 55dd6371283741d4a2f5067a657c317b (127.4.197.129:45317)
I20260929 02:49:30.824203 8479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:46877
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:49:30.849723 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36505
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:31.168584 8674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:31.168885 8674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:31.168999 8674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:31.176294 8674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:31.176409 8674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:31.176559 8674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:31.186363 8674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:31.187680 8674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:31.188968 8674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:31.201653 8679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:31.201565 8680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:31.201615 8682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:31.202420 8674 server_base.cc:1034] running on GCE node
I20260929 02:49:31.203054 8674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:31.203904 8674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:31.205087 8674 hybrid_clock.cc:648] HybridClock initialized: now 1790650171205037 us; error 42 us; skew 500 ppm
I20260929 02:49:31.205363 8674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:31 dist-test-slave-t6c1 krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650171, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:31.208214 8674 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:49:31.209486 8674 webserver.cc:460] Webserver started at http://127.4.197.130:44499/ using document root <none> and password file <none>
I20260929 02:49:31.210060 8674 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:31.210152 8674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:31.210391 8674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:31.213056 8674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "71df88a15ff844eb9f122a472898f400"
format_stamp: "Formatted at 2026-09-29 02:49:31 on dist-test-slave-t6c1"
server_key: "b073f6646dd583236d195bb07300fcd6"
server_key_iv: "c46a733ed288675826cd6c8e9c8d5749"
server_key_version: "encryptionkey@0"
I20260929 02:49:31.213749 8674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "71df88a15ff844eb9f122a472898f400"
format_stamp: "Formatted at 2026-09-29 02:49:31 on dist-test-slave-t6c1"
server_key: "b073f6646dd583236d195bb07300fcd6"
server_key_iv: "c46a733ed288675826cd6c8e9c8d5749"
server_key_version: "encryptionkey@0"
I20260929 02:49:31.218701 8674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:49:31.221824 8689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:31.223151 8674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:31.223287 8674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "71df88a15ff844eb9f122a472898f400"
format_stamp: "Formatted at 2026-09-29 02:49:31 on dist-test-slave-t6c1"
server_key: "b073f6646dd583236d195bb07300fcd6"
server_key_iv: "c46a733ed288675826cd6c8e9c8d5749"
server_key_version: "encryptionkey@0"
I20260929 02:49:31.223455 8674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:31.252585 8674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:31.257054 8674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:31.257299 8674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:31.258222 8674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:31.259542 8674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:31.259615 8674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:31.259706 8674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:31.259755 8674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:31.289671 8674 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:43047
I20260929 02:49:31.289786 8801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:43047 every 8 connection(s)
I20260929 02:49:31.290989 8674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:31.297448 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 8674
I20260929 02:49:31.297609 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:31.298000 4886 external_mini_cluster.cc:1428] Setting key 9a59dc4e47ffa9094733719a592ad6fc
Sep 29 02:49:31 dist-test-slave-t6c1 krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650171, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:31.310797 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.293480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:35141 (local address 127.4.197.190:36505)
0929 02:49:31.293656 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:31.293660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:31.294899 (+ 1239us) server_negotiation.cc:407] Connection header received
0929 02:49:31.296837 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:31.296843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:31.296926 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:31.297022 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:31.298644 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.299466 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.301086 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.301499 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.301619 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:31.305246 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:31.305251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:31.305254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:31.305288 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:31.307802 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.308510 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.308514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.308516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.308643 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.309044 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.309048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.309050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.309310 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:31.309449 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:31.310233 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 02:49:31.310464 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:49:31.311832 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.294024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36505 (local address 127.4.197.130:35141)
0929 02:49:31.294742 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:31.294781 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:31.296558 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:31.297279 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:31.297287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:31.297855 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:31.298474 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:31.298494 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.299606 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.299610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:31.300944 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:31.300957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.301727 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.301734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:31.301960 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:31.302640 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:31.302665 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:31.305020 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:31.307941 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:31.307951 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:31.307968 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:31.308397 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:31.308747 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:31.308751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:31.308755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:31.308941 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:31.309435 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:31.309443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:31.309882 (+ 439us) client_negotiation.cc:769] Sending connection context
0929 02:49:31.310130 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 02:49:31.310674 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":280,"threads_started":1}
I20260929 02:49:31.313369 8802 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36505
I20260929 02:49:31.313733 8802 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
I20260929 02:49:31.314589 8802 heartbeater.cc:511] Master 127.4.197.190:36505 requested a full tablet report, sending...
I20260929 02:49:31.316185 8479 ts_manager.cc:194] Registered new tserver with Master: 71df88a15ff844eb9f122a472898f400 (127.4.197.130:43047)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
I20260929 02:49:31.317148 8479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:35141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:49:31.337837 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36505
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:31.652195 8807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:31.652509 8807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:31.652602 8807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:31.659898 8807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:31.660025 8807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:31.660143 8807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:31.669806 8807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:31.671072 8807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:31.672402 8807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:31.685096 8813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:31.685053 8815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:31.685127 8812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:31.685938 8807 server_base.cc:1034] running on GCE node
I20260929 02:49:31.686592 8807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:31.687613 8807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:31.688838 8807 hybrid_clock.cc:648] HybridClock initialized: now 1790650171688758 us; error 55 us; skew 500 ppm
I20260929 02:49:31.689150 8807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:31 dist-test-slave-t6c1 krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650171, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:31.692160 8807 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:49:31.693552 8807 webserver.cc:460] Webserver started at http://127.4.197.131:33921/ using document root <none> and password file <none>
I20260929 02:49:31.694197 8807 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:31.694307 8807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:31.694576 8807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:31.697335 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "b1c63283bc3d4e32ae1928618554a188"
format_stamp: "Formatted at 2026-09-29 02:49:31 on dist-test-slave-t6c1"
server_key: "07d6249adc77eff20549f8c5d3fb89b3"
server_key_iv: "c783977c2fb17393513cd316c1e3f0a2"
server_key_version: "encryptionkey@0"
I20260929 02:49:31.698063 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "b1c63283bc3d4e32ae1928618554a188"
format_stamp: "Formatted at 2026-09-29 02:49:31 on dist-test-slave-t6c1"
server_key: "07d6249adc77eff20549f8c5d3fb89b3"
server_key_iv: "c783977c2fb17393513cd316c1e3f0a2"
server_key_version: "encryptionkey@0"
I20260929 02:49:31.702740 8807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 02:49:31.705885 8822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:31.707082 8807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:49:31.707218 8807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "b1c63283bc3d4e32ae1928618554a188"
format_stamp: "Formatted at 2026-09-29 02:49:31 on dist-test-slave-t6c1"
server_key: "07d6249adc77eff20549f8c5d3fb89b3"
server_key_iv: "c783977c2fb17393513cd316c1e3f0a2"
server_key_version: "encryptionkey@0"
I20260929 02:49:31.707377 8807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:31.719172 8807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:31.722882 8807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:31.723142 8807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:31.723989 8807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:31.725188 8807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:31.725260 8807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:31.725373 8807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:31.725425 8807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:31.766317 8807 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:33351
I20260929 02:49:31.766367 8934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:33351 every 8 connection(s)
I20260929 02:49:31.767570 8807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:31.768781 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 8807
I20260929 02:49:31.768970 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:31.769374 4886 external_mini_cluster.cc:1428] Setting key 2dfc0eb0f65dc5d82f63d2eff9d1a399
Sep 29 02:49:31 dist-test-slave-t6c1 krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650171, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:31.790803 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.770101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:45349 (local address 127.4.197.190:36505)
0929 02:49:31.770317 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:31.770323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:31.771710 (+ 1387us) server_negotiation.cc:407] Connection header received
0929 02:49:31.773546 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:31.773551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:31.773629 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:31.773710 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:31.775189 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.775922 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.777385 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.777784 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.777887 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:31.785689 (+ 7802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:31.785694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:31.785698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:31.785732 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:31.788176 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.788865 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.788869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.788871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.788987 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.789344 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.789346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.789348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.789613 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:31.789720 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:31.790393 (+ 673us) server_negotiation.cc:299] Negotiation successful
0929 02:49:31.790562 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:49:31.791833 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.770690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36505 (local address 127.4.197.131:45349)
0929 02:49:31.771579 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:31.771616 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:31.773311 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:31.773946 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:31.773954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:31.774457 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:31.775042 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:31.775060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.776054 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.776058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:31.777248 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:31.777268 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.777979 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.777986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:31.778214 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:31.782901 (+ 4687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:31.782953 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:31.785485 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:31.788317 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:31.788324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:31.788338 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:31.788768 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:31.789061 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:31.789064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:31.789066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:31.789242 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:31.789699 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:31.789705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:31.790069 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 02:49:31.790301 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 02:49:31.790693 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":322,"threads_started":1}
I20260929 02:49:31.793134 8935 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36505
I20260929 02:49:31.793401 8935 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:31.794045 8935 heartbeater.cc:511] Master 127.4.197.190:36505 requested a full tablet report, sending...
I20260929 02:49:31.795408 8479 ts_manager.cc:194] Registered new tserver with Master: b1c63283bc3d4e32ae1928618554a188 (127.4.197.131:33351)
I20260929 02:49:31.796237 8479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:45349
I20260929 02:49:31.797119 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:49:31 dist-test-slave-t6c1 krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260929 02:49:31.827447 8668 heartbeater.cc:503] Master 127.4.197.190:36505 was elected leader, sending a full tablet report...
Sep 29 02:49:31 dist-test-slave-t6c1 krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:31.841760 8942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.827491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.4.197.129:45317)
0929 02:49:31.828310 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:31.828317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:31.828342 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:49:31.828395 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:31.828401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:31.828604 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:31.828724 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:31.830127 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.831200 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.832349 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.832922 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.833017 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:31.836472 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:31.836482 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:31.836496 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:31.836531 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:31.839149 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.839820 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.839827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.839830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.839956 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.840262 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.840265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.840267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.840594 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:31.840854 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:31.841196 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 02:49:31.841411 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":495,"threads_started":1}
Sep 29 02:49:31 dist-test-slave-t6c1 krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:31.859040 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.845826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.4.197.190:36505)
0929 02:49:31.846021 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:31.846026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:31.846221 (+ 195us) server_negotiation.cc:407] Connection header received
0929 02:49:31.846334 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:31.846338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:31.846417 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:31.846496 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:31.847650 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.848394 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.849602 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.850035 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.850128 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:31.853871 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:31.853877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:31.853880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:31.853918 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:31.856376 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.857078 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.857083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.857086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.857224 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.857631 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.857635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.857637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.857989 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:31.858176 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:31.858531 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 02:49:31.858697 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 02:49:31.859599 8478 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35548
I20260929 02:49:31.860634 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8540
I20260929 02:49:31.873625 8942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.861454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.4.197.129:45317)
0929 02:49:31.861659 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:31.861665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:31.861852 (+ 187us) server_negotiation.cc:407] Connection header received
0929 02:49:31.861979 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:31.861983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:31.862069 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:31.862145 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:31.863334 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.864177 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.865385 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.865789 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.865917 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:31.868221 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:31.868226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:31.868230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:31.868266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:31.871092 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.871847 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.871852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.871854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.872002 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.872325 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.872328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.872331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.872597 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:31.872831 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:31.873084 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 02:49:31.873278 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:49:31.874307 8663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:31.953786 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 8540
I20260929 02:49:31.966701 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8674
Sep 29 02:49:31 dist-test-slave-t6c1 krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:49:31.981841 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.967586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.4.197.130:43047)
0929 02:49:31.968154 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:31.968161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:31.968187 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:49:31.968246 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:31.968252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:31.968472 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:31.968596 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:31.969701 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.970675 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.971977 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:31.972509 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:31.972602 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:31.976605 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:31.976615 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:31.976628 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:31.976659 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:31.979253 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.979952 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.979960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.979963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.980102 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:31.980499 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:31.980503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:31.980506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:31.980861 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:31.981082 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:31.981307 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 02:49:31.981512 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":271,"threads_started":1}
I20260929 02:49:31.983158 8797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:32.065277 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 8674
I20260929 02:49:32.077484 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8807
Sep 29 02:49:32 dist-test-slave-t6c1 krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:49:32.093900 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:32.078372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.4.197.131:33351)
0929 02:49:32.079040 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:32.079046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:32.079078 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:49:32.079140 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:32.079146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:32.079389 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:32.079526 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:32.080858 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:32.081908 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:32.083499 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:32.084098 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:32.084208 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:32.088603 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:32.088619 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:32.088636 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:32.088683 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:32.091591 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:32.092273 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:32.092279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:32.092282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:32.092405 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:32.092685 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:32.092688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:32.092690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:32.093013 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:32.093252 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:32.093429 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 02:49:32.093622 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":291,"threads_started":1}
I20260929 02:49:32.095168 8930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:32.176441 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 8807
I20260929 02:49:32.188290 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8447
I20260929 02:49:32.189386 8509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:32.290264 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 8447
2026-09-29T02:49:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3659 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:32 dist-test-slave-t6c1 krb5kdc[8954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:32 dist-test-slave-t6c1 krb5kdc[8954](info): set up 2 sockets
Sep 29 02:49:32 dist-test-slave-t6c1 krb5kdc[8954](info): commencing operation
krb5kdc: starting...
W20260929 02:49:33.291219 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:49:33 dist-test-slave-t6c1 krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:49:33.466416 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36515
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:36515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:33.776412 8967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:33.776700 8967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:33.776834 8967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:33.784180 8967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:33.784288 8967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:33.784364 8967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:33.784440 8967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:33.784502 8967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:33.794534 8967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:36515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:33.795851 8967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:33.797155 8967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:33.807797 8973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:33.808175 8975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:33.807832 8972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:33.808583 8967 server_base.cc:1034] running on GCE node
I20260929 02:49:33.809238 8967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:33.810428 8967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:33.811611 8967 hybrid_clock.cc:648] HybridClock initialized: now 1790650173811569 us; error 38 us; skew 500 ppm
I20260929 02:49:33.811914 8967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:33 dist-test-slave-t6c1 krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650173, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:33.814595 8967 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:49:33.815636 8967 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:33.815730 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:33.816027 8967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:33.818657 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "a46e7c3ef2524511974d898b7351d997"
format_stamp: "Formatted at 2026-09-29 02:49:33 on dist-test-slave-t6c1"
server_key: "b354841c08191901e4301e95e3100860"
server_key_iv: "073de351027d09f2e953c815e2c51cb4"
server_key_version: "encryptionkey@0"
I20260929 02:49:33.819396 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "a46e7c3ef2524511974d898b7351d997"
format_stamp: "Formatted at 2026-09-29 02:49:33 on dist-test-slave-t6c1"
server_key: "b354841c08191901e4301e95e3100860"
server_key_iv: "073de351027d09f2e953c815e2c51cb4"
server_key_version: "encryptionkey@0"
I20260929 02:49:33.823921 8967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:49:33.827368 8981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:33.828747 8967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:49:33.828909 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "a46e7c3ef2524511974d898b7351d997"
format_stamp: "Formatted at 2026-09-29 02:49:33 on dist-test-slave-t6c1"
server_key: "b354841c08191901e4301e95e3100860"
server_key_iv: "073de351027d09f2e953c815e2c51cb4"
server_key_version: "encryptionkey@0"
I20260929 02:49:33.829111 8967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:33.847466 8967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:33.851902 8967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:33.852160 8967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:33.869128 8967 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:36515
I20260929 02:49:33.869112 9032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:36515 every 8 connection(s)
I20260929 02:49:33.870041 8967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:33.873353 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 8967
I20260929 02:49:33.873541 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:33.873579 9033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:33.874140 4886 external_mini_cluster.cc:1428] Setting key 997eae362233332bce1a34bfc93a224a
I20260929 02:49:33.886137 9033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997: Bootstrap starting.
Sep 29 02:49:33 dist-test-slave-t6c1 krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:33.889714 9033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:33.890847 9033 log.cc:824] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:33.893522 9033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997: No bootstrap required, opened a new log
I20260929 02:49:33.896740 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:33.875995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.4.197.190:36515)
0929 02:49:33.876761 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:33.876773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:33.876816 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:49:33.879047 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:33.879069 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:33.879453 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:33.879778 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:33.881472 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:33.882757 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:33.884504 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:33.885143 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:33.885274 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:33.889651 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:33.889659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:33.889673 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:33.889710 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:33.892872 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:33.893522 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:33.893528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:33.893533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:33.893717 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:33.894062 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:33.894066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:33.894068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:33.894584 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:33.894857 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:33.895363 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 02:49:33.895756 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":364,"threads_started":1}
I20260929 02:49:33.901527 9033 raft_consensus.cc:348] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46e7c3ef2524511974d898b7351d997" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36515 } }
I20260929 02:49:33.901906 9033 raft_consensus.cc:374] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:33.901995 9033 raft_consensus.cc:729] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46e7c3ef2524511974d898b7351d997, State: Initialized, Role: FOLLOWER
I20260929 02:49:33.902535 9033 consensus_queue.cc:260] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46e7c3ef2524511974d898b7351d997" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36515 } }
I20260929 02:49:33.902738 9033 raft_consensus.cc:388] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:33.902869 9033 raft_consensus.cc:482] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:33.903059 9033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:33.905417 9033 raft_consensus.cc:504] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46e7c3ef2524511974d898b7351d997" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36515 } }
I20260929 02:49:33.905853 9033 leader_election.cc:302] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a46e7c3ef2524511974d898b7351d997; no voters:
I20260929 02:49:33.906323 9033 leader_election.cc:288] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:33.906452 9038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:33.907083 9038 raft_consensus.cc:686] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [term 1 LEADER]: Becoming Leader. State: Replica: a46e7c3ef2524511974d898b7351d997, State: Running, Role: LEADER
I20260929 02:49:33.907572 9038 consensus_queue.cc:237] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46e7c3ef2524511974d898b7351d997" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36515 } }
I20260929 02:49:33.908260 9033 sys_catalog.cc:564] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:33.909981 9040 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a46e7c3ef2524511974d898b7351d997. Latest consensus state: current_term: 1 leader_uuid: "a46e7c3ef2524511974d898b7351d997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46e7c3ef2524511974d898b7351d997" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36515 } } }
I20260929 02:49:33.910400 9040 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:33.913604 9039 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a46e7c3ef2524511974d898b7351d997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46e7c3ef2524511974d898b7351d997" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36515 } } }
I20260929 02:49:33.913913 9039 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:33.914650 9046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:33.919728 9046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:33.928112 9046 catalog_manager.cc:1347] Generated new cluster ID: 612dfdde8e894838b27e780508e7b9a5
I20260929 02:49:33.928239 9046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:33.940449 9046 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:33.942214 9046 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:33.960747 9046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997: Generated new TSK 0
I20260929 02:49:33.961923 9046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:49:34.004426 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36515
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:34.315286 9059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:34.315572 9059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:34.315667 9059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:34.322539 9059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:34.322636 9059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:34.322749 9059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:34.332314 9059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:34.333644 9059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:34.335073 9059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:34.347630 9064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:34.347630 9065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:34.347681 9067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:34.348506 9059 server_base.cc:1034] running on GCE node
I20260929 02:49:34.349057 9059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:34.350054 9059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:34.351286 9059 hybrid_clock.cc:648] HybridClock initialized: now 1790650174351240 us; error 34 us; skew 500 ppm
I20260929 02:49:34.351584 9059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:34 dist-test-slave-t6c1 krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650174, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:34.354435 9059 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:49:34.355530 9059 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:34.355626 9059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:34.355953 9059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:34.358809 9059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "3ec09ef81e02497aa853bae1704cf0a4"
format_stamp: "Formatted at 2026-09-29 02:49:34 on dist-test-slave-t6c1"
server_key: "d63f51ada7d02dace728eca52106427a"
server_key_iv: "49fd8f7fa50f0d9d3aa4c763ccfdfa71"
server_key_version: "encryptionkey@0"
I20260929 02:49:34.359604 9059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "3ec09ef81e02497aa853bae1704cf0a4"
format_stamp: "Formatted at 2026-09-29 02:49:34 on dist-test-slave-t6c1"
server_key: "d63f51ada7d02dace728eca52106427a"
server_key_iv: "49fd8f7fa50f0d9d3aa4c763ccfdfa71"
server_key_version: "encryptionkey@0"
I20260929 02:49:34.364332 9059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:49:34.367709 9073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:34.369262 9059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:34.369431 9059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "3ec09ef81e02497aa853bae1704cf0a4"
format_stamp: "Formatted at 2026-09-29 02:49:34 on dist-test-slave-t6c1"
server_key: "d63f51ada7d02dace728eca52106427a"
server_key_iv: "49fd8f7fa50f0d9d3aa4c763ccfdfa71"
server_key_version: "encryptionkey@0"
I20260929 02:49:34.369637 9059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:34.395360 9059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:34.399433 9059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:34.399683 9059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:34.400585 9059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:34.401847 9059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:34.401926 9059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:34.402021 9059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:34.402089 9059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:34.429555 9059 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:34009
I20260929 02:49:34.429605 9185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:34009 every 8 connection(s)
I20260929 02:49:34.430371 9059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:34.431631 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 9059
I20260929 02:49:34.431831 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:34.432489 4886 external_mini_cluster.cc:1428] Setting key fc157b878dfa0786cd02c68f0b2c6850
Sep 29 02:49:34 dist-test-slave-t6c1 krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650174, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:49:34.452622 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:34.433043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:49483 (local address 127.4.197.190:36515)
0929 02:49:34.433544 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:34.433551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:34.434678 (+ 1127us) server_negotiation.cc:407] Connection header received
0929 02:49:34.437198 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:34.437202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:34.437267 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:34.437371 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:34.439251 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:34.439973 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:34.441726 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:34.442317 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:34.442452 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:34.445863 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:34.445869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:34.445873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:34.445910 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:34.449165 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:34.449948 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:34.449954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:34.449958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:34.450109 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:34.450632 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:34.450637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:34.450641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:34.451063 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:34.451220 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:34.452009 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 02:49:34.452248 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":268,"threads_started":1}
I20260929 02:49:34.453675 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:34.433696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36515 (local address 127.4.197.129:49483)
0929 02:49:34.434542 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:34.434583 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:34.436887 (+ 2304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:34.437627 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:34.437638 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:34.438318 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:34.439099 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:34.439126 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:34.440122 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:34.440128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:34.441574 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:34.441586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:34.442538 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:34.442543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:34.442737 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:34.443403 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:34.443425 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:34.445628 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:34.449333 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:34.449340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:34.449355 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:34.449815 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:34.450254 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:34.450259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:34.450261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:34.450501 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:34.451204 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:34.451210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:34.451590 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 02:49:34.451859 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 02:49:34.452474 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":324,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
I20260929 02:49:34.455430 9186 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36515
I20260929 02:49:34.455730 9186 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:34.456389 9186 heartbeater.cc:511] Master 127.4.197.190:36515 requested a full tablet report, sending...
I20260929 02:49:34.458635 8998 ts_manager.cc:194] Registered new tserver with Master: 3ec09ef81e02497aa853bae1704cf0a4 (127.4.197.129:34009)
I20260929 02:49:34.460263 8998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:49483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:49:34.477559 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36515
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:34.794117 9192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:34.794394 9192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:34.794490 9192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:34.801730 9192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:34.801873 9192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:34.801992 9192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:34.812134 9192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:34.813417 9192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:34.815008 9192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:34.827615 9197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:34.827579 9198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:34.827669 9200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:34.828332 9192 server_base.cc:1034] running on GCE node
I20260929 02:49:34.828914 9192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:34.829823 9192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:34.831019 9192 hybrid_clock.cc:648] HybridClock initialized: now 1790650174830961 us; error 48 us; skew 500 ppm
I20260929 02:49:34.831312 9192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:34 dist-test-slave-t6c1 krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650174, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:34.834390 9192 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:49:34.835529 9192 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:34.835618 9192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:34.835875 9192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:34.838775 9192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "3b6cf07de92f472d87883cda901c6106"
format_stamp: "Formatted at 2026-09-29 02:49:34 on dist-test-slave-t6c1"
server_key: "a47af740e789dcbf80a11d519354468b"
server_key_iv: "db3a96e8f9c35b2e92832dc54021b992"
server_key_version: "encryptionkey@0"
I20260929 02:49:34.839612 9192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "3b6cf07de92f472d87883cda901c6106"
format_stamp: "Formatted at 2026-09-29 02:49:34 on dist-test-slave-t6c1"
server_key: "a47af740e789dcbf80a11d519354468b"
server_key_iv: "db3a96e8f9c35b2e92832dc54021b992"
server_key_version: "encryptionkey@0"
I20260929 02:49:34.844563 9192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:49:34.848059 9206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:34.849404 9192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:34.849555 9192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "3b6cf07de92f472d87883cda901c6106"
format_stamp: "Formatted at 2026-09-29 02:49:34 on dist-test-slave-t6c1"
server_key: "a47af740e789dcbf80a11d519354468b"
server_key_iv: "db3a96e8f9c35b2e92832dc54021b992"
server_key_version: "encryptionkey@0"
I20260929 02:49:34.849740 9192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:34.876787 9192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:34.881114 9192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:34.881376 9192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:34.882330 9192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:34.883682 9192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:34.883767 9192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:34.883862 9192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:34.883924 9192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:34.912726 9192 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:43653
I20260929 02:49:34.912776 9318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:43653 every 8 connection(s)
I20260929 02:49:34.913514 9192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:34.915649 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 9192
I20260929 02:49:34.915820 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:34.916260 4886 external_mini_cluster.cc:1428] Setting key 8e50dd6acda3f695aa8b377bb97e6ca1
Sep 29 02:49:34 dist-test-slave-t6c1 krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650174, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:34.933477 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:34.915835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:48995 (local address 127.4.197.190:36515)
0929 02:49:34.916031 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:34.916037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:34.917329 (+ 1292us) server_negotiation.cc:407] Connection header received
0929 02:49:34.919569 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:34.919573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:34.919636 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:34.919751 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:34.921520 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:34.922232 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:34.924286 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:34.924681 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:34.924792 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:34.928079 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:34.928084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:34.928087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:34.928117 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:34.930443 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:34.931228 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:34.931232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:34.931235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:34.931363 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:34.931759 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:34.931763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:34.931765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:34.932055 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:34.932180 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:34.932913 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 02:49:34.933133 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
I20260929 02:49:34.934480 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:34.916422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36515 (local address 127.4.197.130:48995)
0929 02:49:34.917185 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:34.917234 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:34.919275 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:34.919978 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:34.919990 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:34.920641 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:34.921336 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:34.921358 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:34.922383 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:34.922388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:34.924157 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:34.924169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:34.924887 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:34.924894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:34.925128 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:34.925744 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:34.925766 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:34.927864 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:34.930591 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:34.930601 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:34.930619 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:34.931110 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:34.931467 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:34.931472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:34.931476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:34.931652 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:34.932171 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:34.932179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:34.932553 (+ 374us) client_negotiation.cc:769] Sending connection context
0929 02:49:34.932797 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 02:49:34.933309 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":294,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
I20260929 02:49:34.935930 9319 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36515
I20260929 02:49:34.936201 9319 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:34.936820 9319 heartbeater.cc:511] Master 127.4.197.190:36515 requested a full tablet report, sending...
I20260929 02:49:34.938230 8998 ts_manager.cc:194] Registered new tserver with Master: 3b6cf07de92f472d87883cda901c6106 (127.4.197.130:43653)
I20260929 02:49:34.939280 8998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:48995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:49:34.957614 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36515
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:35.272059 9324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:35.272398 9324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:35.272497 9324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:35.280090 9324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:35.280237 9324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:35.280364 9324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:35.290472 9324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:35.291914 9324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:35.293354 9324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:35.306514 9332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:35.306631 9330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:35.306855 9329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:35.307507 9324 server_base.cc:1034] running on GCE node
I20260929 02:49:35.308079 9324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:35.308940 9324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:35.310140 9324 hybrid_clock.cc:648] HybridClock initialized: now 1790650175310079 us; error 55 us; skew 500 ppm
I20260929 02:49:35.310427 9324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:35 dist-test-slave-t6c1 krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650175, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:35.313431 9324 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:49:35.314546 9324 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:35.314656 9324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:35.315037 9324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:35.318018 9324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "d5ebc8f7eecd4676aaaf9a042a86e923"
format_stamp: "Formatted at 2026-09-29 02:49:35 on dist-test-slave-t6c1"
server_key: "104416d059a498ce899c30d860df0052"
server_key_iv: "9e03f086438a6fdbf17ca13fe92de62b"
server_key_version: "encryptionkey@0"
I20260929 02:49:35.318854 9324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "d5ebc8f7eecd4676aaaf9a042a86e923"
format_stamp: "Formatted at 2026-09-29 02:49:35 on dist-test-slave-t6c1"
server_key: "104416d059a498ce899c30d860df0052"
server_key_iv: "9e03f086438a6fdbf17ca13fe92de62b"
server_key_version: "encryptionkey@0"
I20260929 02:49:35.323776 9324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:49:35.327097 9338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:35.328382 9324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:49:35.328531 9324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "d5ebc8f7eecd4676aaaf9a042a86e923"
format_stamp: "Formatted at 2026-09-29 02:49:35 on dist-test-slave-t6c1"
server_key: "104416d059a498ce899c30d860df0052"
server_key_iv: "9e03f086438a6fdbf17ca13fe92de62b"
server_key_version: "encryptionkey@0"
I20260929 02:49:35.328719 9324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:35.352221 9324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:35.356606 9324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:35.356917 9324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:35.357920 9324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:35.359225 9324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:35.359313 9324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:35.359390 9324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:35.359440 9324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:35.387923 9324 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:35959
I20260929 02:49:35.387964 9450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:35959 every 8 connection(s)
I20260929 02:49:35.388777 9324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:35.396260 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 9324
I20260929 02:49:35.396448 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:35.396843 4886 external_mini_cluster.cc:1428] Setting key 3a6e3cfa738eb2e4a3b61af24af52a78
Sep 29 02:49:35 dist-test-slave-t6c1 krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650175, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:35.407099 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.391094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:55099 (local address 127.4.197.190:36515)
0929 02:49:35.391291 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:35.391297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:35.392437 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 02:49:35.394211 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:35.394216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:35.394300 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:35.394396 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:35.396036 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.396864 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.398665 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.399137 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.399250 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.402322 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:35.402327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:35.402330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:35.402362 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:35.404563 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:35.405209 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:35.405213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:35.405215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:35.405339 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:35.405724 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:35.405727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:35.405730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:35.405992 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:35.406096 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:35.406656 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 02:49:35.406810 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:49:35.408031 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.391621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36515 (local address 127.4.197.131:55099)
0929 02:49:35.392285 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:35.392323 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:35.393952 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:35.394633 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:35.394642 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:35.395239 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:35.395854 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.395882 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.397030 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.397033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.398531 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.398543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.399351 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.399358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.399586 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.400159 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:35.400181 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:35.402131 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:35.404682 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:35.404690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:35.404706 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:35.405098 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:35.405452 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:35.405455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:35.405458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:35.405631 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:35.406079 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:35.406084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:35.406386 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 02:49:35.406599 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 02:49:35.407004 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":270,"threads_started":1}
I20260929 02:49:35.409260 9451 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36515
I20260929 02:49:35.409525 9451 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:35.410109 9451 heartbeater.cc:511] Master 127.4.197.190:36515 requested a full tablet report, sending...
I20260929 02:49:35.411526 8998 ts_manager.cc:194] Registered new tserver with Master: d5ebc8f7eecd4676aaaf9a042a86e923 (127.4.197.131:35959)
I20260929 02:49:35.412042 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:35.412397 8998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:55099
I20260929 02:49:35.427687 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.416902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.4.197.190:36515)
0929 02:49:35.417046 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:35.417051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:35.417523 (+ 472us) server_negotiation.cc:407] Connection header received
0929 02:49:35.417649 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:35.417652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:35.417707 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:35.417779 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:35.419099 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.419782 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.420893 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.421273 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.421364 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.423438 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:35.423442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:35.423445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:35.423473 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:35.425715 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:35.426208 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:35.426211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:35.426213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:35.426343 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:35.426612 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:35.426615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:35.426617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:35.426888 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:35.427082 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:35.427301 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 02:49:35.427462 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:49:35.431313 8998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:35.438836 8998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:35.462471 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.451729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.4.197.130:43653)
0929 02:49:35.454691 (+ 2962us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:35.454701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:35.454730 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:49:35.454826 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:35.454834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:35.455155 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:35.455351 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:35.455981 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.457398 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.459998 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.461334 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.461647 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.461843 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:35.461974 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 02:49:35.462142 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2794,"thread_start_us":2643,"threads_started":1}
I20260929 02:49:35.462500 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.452561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:34009 (local address 127.0.0.1:39866)
0929 02:49:35.454053 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:35.454073 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:35.454216 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:35.455681 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:35.455686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:35.455713 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:35.456154 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.456164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.457890 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.457895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.459647 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.459658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.461391 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.461398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.461585 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.461664 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 02:49:35.461802 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 02:49:35.461990 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1311,"mutex_wait_us":22,"thread_start_us":1887,"threads_started":1}
I20260929 02:49:35.462858 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.451222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:35959 (local address 127.0.0.1:57922)
0929 02:49:35.452131 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:35.452177 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:35.452335 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:35.455917 (+ 3582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:35.455921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:35.455942 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:35.456353 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.456363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.458293 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.458298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.460373 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.460386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.462085 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.462091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.462260 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.462294 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:49:35.462365 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 02:49:35.462506 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":5,"thread_start_us":530,"threads_started":1}
I20260929 02:49:35.463001 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.451261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:43653 (local address 127.0.0.1:43238)
0929 02:49:35.452803 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:35.452825 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:35.452979 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:35.455369 (+ 2390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:35.455377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:35.455404 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:35.455797 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.455811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.457542 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.457547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.459850 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.459864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.461563 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.461570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.461804 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.461834 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:49:35.461932 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:49:35.462063 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1442,"mutex_wait_us":42,"thread_start_us":1845,"threads_started":1}
I20260929 02:49:35.463588 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.453108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.4.197.129:34009)
0929 02:49:35.455062 (+ 1954us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:35.455071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:35.455097 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:49:35.455195 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:35.455203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:35.455441 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:35.455753 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:35.456346 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.457759 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.459791 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.461180 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.461319 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.461563 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:35.462503 (+ 940us) server_negotiation.cc:299] Negotiation successful
0929 02:49:35.463255 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1729,"thread_start_us":1590,"threads_started":1}
I20260929 02:49:35.463584 9186 heartbeater.cc:503] Master 127.4.197.190:36515 was elected leader, sending a full tablet report...
I20260929 02:49:35.463953 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.451314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.4.197.131:35959)
0929 02:49:35.455372 (+ 4058us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:35.455380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:35.455413 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:49:35.455500 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:35.455506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:35.455708 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:35.456508 (+ 800us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:35.456801 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.458154 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.460541 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.461878 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.463194 (+ 1316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.463433 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:35.463499 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 02:49:35.463651 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3899,"thread_start_us":3736,"threads_started":1}
I20260929 02:49:35.467211 9121 tablet_service.cc:1467] Processing CreateTablet for tablet ed4c20206d4f406986d9e63970c175f2 (DEFAULT_TABLE table=test-table [id=64b506e20952455685086a5004187085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:35.467378 9386 tablet_service.cc:1467] Processing CreateTablet for tablet ed4c20206d4f406986d9e63970c175f2 (DEFAULT_TABLE table=test-table [id=64b506e20952455685086a5004187085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:35.467378 9254 tablet_service.cc:1467] Processing CreateTablet for tablet ed4c20206d4f406986d9e63970c175f2 (DEFAULT_TABLE table=test-table [id=64b506e20952455685086a5004187085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:35.468384 9121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4c20206d4f406986d9e63970c175f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:35.468637 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4c20206d4f406986d9e63970c175f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:35.468638 9254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4c20206d4f406986d9e63970c175f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:35.479743 9469 tablet_bootstrap.cc:492] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: Bootstrap starting.
I20260929 02:49:35.483589 9471 tablet_bootstrap.cc:492] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: Bootstrap starting.
I20260929 02:49:35.483932 9469 tablet_bootstrap.cc:654] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:35.485213 9470 tablet_bootstrap.cc:492] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: Bootstrap starting.
I20260929 02:49:35.485416 9469 log.cc:824] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:35.487084 9471 tablet_bootstrap.cc:654] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:35.488265 9470 tablet_bootstrap.cc:654] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:35.488298 9471 log.cc:824] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:35.489012 9469 tablet_bootstrap.cc:492] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: No bootstrap required, opened a new log
I20260929 02:49:35.489254 9470 log.cc:824] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:35.489342 9469 ts_tablet_manager.cc:1397] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 02:49:35.492221 9471 tablet_bootstrap.cc:492] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: No bootstrap required, opened a new log
I20260929 02:49:35.492548 9471 ts_tablet_manager.cc:1397] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260929 02:49:35.492671 9470 tablet_bootstrap.cc:492] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: No bootstrap required, opened a new log
I20260929 02:49:35.492987 9470 ts_tablet_manager.cc:1397] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260929 02:49:35.499819 9469 raft_consensus.cc:348] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 35959 } } peers { permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 } } peers { permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 } }
I20260929 02:49:35.500463 9469 raft_consensus.cc:374] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:35.500612 9469 raft_consensus.cc:729] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec09ef81e02497aa853bae1704cf0a4, State: Initialized, Role: FOLLOWER
I20260929 02:49:35.500805 9470 raft_consensus.cc:348] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 35959 } } peers { permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 } } peers { permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 } }
I20260929 02:49:35.501173 9470 raft_consensus.cc:374] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:35.501266 9470 raft_consensus.cc:729] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6cf07de92f472d87883cda901c6106, State: Initialized, Role: FOLLOWER
I20260929 02:49:35.501303 9469 consensus_queue.cc:260] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 35959 } } peers { permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 } } peers { permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 } }
I20260929 02:49:35.501785 9470 consensus_queue.cc:260] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 35959 } } peers { permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 } } peers { permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 } }
I20260929 02:49:35.502503 9471 raft_consensus.cc:348] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 35959 } } peers { permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 } } peers { permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 } }
I20260929 02:49:35.502971 9471 raft_consensus.cc:374] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:35.503113 9471 raft_consensus.cc:729] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ebc8f7eecd4676aaaf9a042a86e923, State: Initialized, Role: FOLLOWER
I20260929 02:49:35.503259 9469 ts_tablet_manager.cc:1428] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260929 02:49:35.503799 9471 consensus_queue.cc:260] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 35959 } } peers { permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 } } peers { permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 } }
I20260929 02:49:35.504160 9319 heartbeater.cc:503] Master 127.4.197.190:36515 was elected leader, sending a full tablet report...
I20260929 02:49:35.504770 9470 ts_tablet_manager.cc:1428] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260929 02:49:35.505728 9451 heartbeater.cc:503] Master 127.4.197.190:36515 was elected leader, sending a full tablet report...
I20260929 02:49:35.507280 9471 ts_tablet_manager.cc:1428] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
W20260929 02:49:35.640206 9452 tablet.cc:2369] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:49:35.665251 9320 tablet.cc:2369] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:49:35.682708 9187 tablet.cc:2369] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:35.700508 9477 raft_consensus.cc:482] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:35.700806 9477 raft_consensus.cc:504] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 35959 } } peers { permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 } } peers { permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 } }
I20260929 02:49:35.702407 9477 leader_election.cc:288] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6cf07de92f472d87883cda901c6106 (127.4.197.130:43653), 3ec09ef81e02497aa853bae1704cf0a4 (127.4.197.129:34009)
I20260929 02:49:35.707515 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.702746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:60117 (local address 127.4.197.130:43653)
0929 02:49:35.702993 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:35.702999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:35.703019 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:49:35.703095 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:35.703100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:35.703176 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:35.703296 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:35.703718 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.704420 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.705958 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.706700 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.706831 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.706899 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:35.707162 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 02:49:35.707289 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:49:35.707520 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.702625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:43653 (local address 127.4.197.131:60117)
0929 02:49:35.702832 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:35.702850 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:35.702978 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:35.703315 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:35.703320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:35.703349 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:35.703569 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.703580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.704544 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.704548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.705807 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.705821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.706841 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.706846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.707029 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.707052 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:49:35.707129 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:49:35.707252 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":1}
I20260929 02:49:35.708428 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.702625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:34009 (local address 127.4.197.131:56221)
0929 02:49:35.703681 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:35.703696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:35.703796 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:35.704061 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:35.704065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:35.704088 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:35.704356 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.704364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.705330 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.705335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.706649 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:35.706660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.707826 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.707831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:35.707980 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.708010 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:49:35.708084 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:49:35.708165 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":430,"thread_start_us":302,"threads_started":1}
I20260929 02:49:35.708498 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.702971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:56221 (local address 127.4.197.129:34009)
0929 02:49:35.703148 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:35.703152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:35.703753 (+ 601us) server_negotiation.cc:407] Connection header received
0929 02:49:35.703913 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:35.703917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:35.703970 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:35.704040 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:35.704497 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.705194 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.706786 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.707556 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.707714 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.707772 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:35.708132 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 02:49:35.708253 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:49:35.708498 9274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4c20206d4f406986d9e63970c175f2" candidate_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6cf07de92f472d87883cda901c6106" is_pre_election: true
I20260929 02:49:35.709028 9274 raft_consensus.cc:2454] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ebc8f7eecd4676aaaf9a042a86e923 in term 0.
I20260929 02:49:35.709400 9141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4c20206d4f406986d9e63970c175f2" candidate_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec09ef81e02497aa853bae1704cf0a4" is_pre_election: true
I20260929 02:49:35.709676 9342 leader_election.cc:302] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6cf07de92f472d87883cda901c6106, d5ebc8f7eecd4676aaaf9a042a86e923; no voters:
I20260929 02:49:35.709853 9141 raft_consensus.cc:2454] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ebc8f7eecd4676aaaf9a042a86e923 in term 0.
I20260929 02:49:35.710011 9477 raft_consensus.cc:2781] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:35.710140 9477 raft_consensus.cc:482] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:35.710239 9477 raft_consensus.cc:3037] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:35.712349 9477 raft_consensus.cc:504] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 35959 } } peers { permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 } } peers { permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 } }
I20260929 02:49:35.712839 9477 leader_election.cc:288] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6cf07de92f472d87883cda901c6106 (127.4.197.130:43653), 3ec09ef81e02497aa853bae1704cf0a4 (127.4.197.129:34009)
I20260929 02:49:35.713176 9274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4c20206d4f406986d9e63970c175f2" candidate_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6cf07de92f472d87883cda901c6106"
I20260929 02:49:35.713332 9274 raft_consensus.cc:3037] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:35.713338 9141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4c20206d4f406986d9e63970c175f2" candidate_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec09ef81e02497aa853bae1704cf0a4"
I20260929 02:49:35.713527 9141 raft_consensus.cc:3037] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:35.715565 9274 raft_consensus.cc:2454] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ebc8f7eecd4676aaaf9a042a86e923 in term 1.
I20260929 02:49:35.715873 9342 leader_election.cc:302] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6cf07de92f472d87883cda901c6106, d5ebc8f7eecd4676aaaf9a042a86e923; no voters:
I20260929 02:49:35.716099 9477 raft_consensus.cc:2781] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:35.716187 9141 raft_consensus.cc:2454] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ebc8f7eecd4676aaaf9a042a86e923 in term 1.
I20260929 02:49:35.716497 9477 raft_consensus.cc:686] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 1 LEADER]: Becoming Leader. State: Replica: d5ebc8f7eecd4676aaaf9a042a86e923, State: Running, Role: LEADER
I20260929 02:49:35.716955 9477 consensus_queue.cc:237] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 35959 } } peers { permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 } } peers { permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 } }
I20260929 02:49:35.721318 8996 catalog_manager.cc:5495] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5ebc8f7eecd4676aaaf9a042a86e923 (127.4.197.131). New cstate: current_term: 1 leader_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebc8f7eecd4676aaaf9a042a86e923" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 35959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 } health_report { overall_health: UNKNOWN } } }
I20260929 02:49:35.764807 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.759979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.4.197.131:35959)
0929 02:49:35.760150 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:35.760155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:35.760173 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:49:35.760261 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:35.760264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:35.760359 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:35.760479 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:35.760837 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.761557 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.762822 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.763266 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.763366 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.763838 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:35.764381 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:35.764504 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:35.764540 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 02:49:35.764608 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:49:35.772768 9274 raft_consensus.cc:1261] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106 [term 1 FOLLOWER]: Refusing update from remote peer d5ebc8f7eecd4676aaaf9a042a86e923: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:35.772763 9141 raft_consensus.cc:1261] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4 [term 1 FOLLOWER]: Refusing update from remote peer d5ebc8f7eecd4676aaaf9a042a86e923: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:35.773967 9479 consensus_queue.cc:1035] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6cf07de92f472d87883cda901c6106" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:35.774312 9477 consensus_queue.cc:1035] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec09ef81e02497aa853bae1704cf0a4" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:35.799835 8996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:35.800101 8996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:35.803541 8996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a46e7c3ef2524511974d898b7351d997: Sending DeleteTablet for 3 replicas of tablet ed4c20206d4f406986d9e63970c175f2
I20260929 02:49:35.804610 9386 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4c20206d4f406986d9e63970c175f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:35 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:57922
I20260929 02:49:35.804589 9254 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4c20206d4f406986d9e63970c175f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:35 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:43238
I20260929 02:49:35.804849 9121 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4c20206d4f406986d9e63970c175f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:35 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:39866
I20260929 02:49:35.806744 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 9059
I20260929 02:49:35.823467 9494 tablet_replica.cc:330] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: stopping tablet replica
I20260929 02:49:35.823601 9492 tablet_replica.cc:330] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: stopping tablet replica
I20260929 02:49:35.824128 9493 tablet_replica.cc:330] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: stopping tablet replica
I20260929 02:49:35.824102 9492 raft_consensus.cc:2229] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:35.824092 9494 raft_consensus.cc:2229] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:35.824551 9493 raft_consensus.cc:2229] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:35.824637 9494 raft_consensus.cc:2258] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:35.825033 9493 raft_consensus.cc:2258] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:35.825222 9492 raft_consensus.cc:2258] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:35.827523 9493 ts_tablet_manager.cc:1905] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:35.827571 9494 ts_tablet_manager.cc:1905] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:49:35 dist-test-slave-t6c1 krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:35.833218 9492 ts_tablet_manager.cc:1905] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:35.834213 9493 ts_tablet_manager.cc:1918] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:35.834398 9493 log.cc:1196] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/ed4c20206d4f406986d9e63970c175f2
I20260929 02:49:35.834894 9493 ts_tablet_manager.cc:1939] T ed4c20206d4f406986d9e63970c175f2 P 3b6cf07de92f472d87883cda901c6106: Deleting consensus metadata
I20260929 02:49:35.835901 9494 ts_tablet_manager.cc:1918] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:35.836127 9494 log.cc:1196] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/ed4c20206d4f406986d9e63970c175f2
I20260929 02:49:35.836647 8985 catalog_manager.cc:4841] TS 3b6cf07de92f472d87883cda901c6106 (127.4.197.130:43653): tablet ed4c20206d4f406986d9e63970c175f2 (table test-table [id=64b506e20952455685086a5004187085]) successfully deleted
I20260929 02:49:35.837488 9494 ts_tablet_manager.cc:1939] T ed4c20206d4f406986d9e63970c175f2 P 3ec09ef81e02497aa853bae1704cf0a4: Deleting consensus metadata
I20260929 02:49:35.839169 8983 catalog_manager.cc:4841] TS 3ec09ef81e02497aa853bae1704cf0a4 (127.4.197.129:34009): tablet ed4c20206d4f406986d9e63970c175f2 (table test-table [id=64b506e20952455685086a5004187085]) successfully deleted
I20260929 02:49:35.839525 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.808350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.4.197.129:34009)
0929 02:49:35.808516 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:35.808522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:35.813513 (+ 4991us) server_negotiation.cc:407] Connection header received
0929 02:49:35.814336 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:35.814344 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:35.814436 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:35.814540 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:35.818219 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.819572 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.825205 (+ 5633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.825864 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.825998 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.831433 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:35.831445 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:35.831461 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:35.831502 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:35.835507 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:35.836408 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:35.836417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:35.836421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:35.836588 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:35.837045 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:35.837051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:35.837054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:35.837478 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:35.837783 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:35.838934 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0929 02:49:35.839137 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:49:35.839932 9492 ts_tablet_manager.cc:1918] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:35.840102 9492 log.cc:1196] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/ed4c20206d4f406986d9e63970c175f2
I20260929 02:49:35.840245 9181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:35.840584 9492 ts_tablet_manager.cc:1939] T ed4c20206d4f406986d9e63970c175f2 P d5ebc8f7eecd4676aaaf9a042a86e923: Deleting consensus metadata
I20260929 02:49:35.842228 8983 catalog_manager.cc:4841] TS d5ebc8f7eecd4676aaaf9a042a86e923 (127.4.197.131:35959): tablet ed4c20206d4f406986d9e63970c175f2 (table test-table [id=64b506e20952455685086a5004187085]) successfully deleted
I20260929 02:49:35.955384 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 9059
I20260929 02:49:35.968446 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 9192
Sep 29 02:49:35 dist-test-slave-t6c1 krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:49:35.985302 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.969274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.4.197.130:43653)
0929 02:49:35.969438 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:35.969442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:35.969460 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:49:35.969651 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:35.969654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:35.969717 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:35.969827 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:35.971540 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.972265 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.974036 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:35.974406 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:35.974500 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:35.979472 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:35.979487 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:35.979503 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:35.979543 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:35.982324 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:35.983162 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:35.983170 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:35.983173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:35.983304 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:35.983698 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:35.983703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:35.983705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:35.984109 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:35.984390 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:35.984725 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 02:49:35.984975 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:49:35.986078 9314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:36.073098 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 9192
I20260929 02:49:36.085804 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 9324
Sep 29 02:49:36 dist-test-slave-t6c1 krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:49:36.102965 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:36.086647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.4.197.131:35959)
0929 02:49:36.086870 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:36.086877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:36.086902 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:49:36.087010 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:36.087017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:36.087105 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:36.087249 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:36.088399 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:36.089640 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:36.090815 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:36.091699 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:36.091849 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:36.095912 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:36.095929 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:36.095949 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:36.096004 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:36.099830 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:36.100536 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:36.100546 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:36.100549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:36.100757 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:36.101104 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:36.101109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:36.101113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:36.101588 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:36.101965 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:36.102236 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 02:49:36.102491 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:49:36.103777 9446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:36.220865 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 9324
I20260929 02:49:36.235508 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 8967
I20260929 02:49:36.236539 9028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:36.341063 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 8967
2026-09-29T02:49:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4051 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T02:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:36Z Disabled control of system clock
I20260929 02:49:36.400180 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33395
--webserver_interface=127.4.197.190
--webserver_port=0
--builtin_ntp_servers=127.4.197.148:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:33395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:49:36.711127 9503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:36.711401 9503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:36.711493 9503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:36.718614 9503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:36.718707 9503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:36.718750 9503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:36.718806 9503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:36.718849 9503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:36.727677 9503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:33395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:36.728852 9503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:36.730666 9503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:36.742148 9508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:36.742112 9509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:36.742177 9511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:36.743096 9503 server_base.cc:1034] running on GCE node
I20260929 02:49:36.743928 9503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:36.745438 9503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:36.746635 9503 hybrid_clock.cc:648] HybridClock initialized: now 1790650176746596 us; error 37 us; skew 500 ppm
I20260929 02:49:36.747022 9503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:49:36.749348 9503 webserver.cc:460] Webserver started at http://127.4.197.190:40623/ using document root <none> and password file <none>
I20260929 02:49:36.750043 9503 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:36.750135 9503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:36.750444 9503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:36.753381 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "8912ccb28df541e69ad689b93e4145e0"
format_stamp: "Formatted at 2026-09-29 02:49:36 on dist-test-slave-t6c1"
server_key: "24d82788c5af9260ec2cb7c61551884a"
server_key_iv: "a290171ba20966ce476142cfdfe6a087"
server_key_version: "encryptionkey@0"
I20260929 02:49:36.754172 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "8912ccb28df541e69ad689b93e4145e0"
format_stamp: "Formatted at 2026-09-29 02:49:36 on dist-test-slave-t6c1"
server_key: "24d82788c5af9260ec2cb7c61551884a"
server_key_iv: "a290171ba20966ce476142cfdfe6a087"
server_key_version: "encryptionkey@0"
I20260929 02:49:36.759354 9503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:49:36.762742 9517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:36.764132 9503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:36.764282 9503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "8912ccb28df541e69ad689b93e4145e0"
format_stamp: "Formatted at 2026-09-29 02:49:36 on dist-test-slave-t6c1"
server_key: "24d82788c5af9260ec2cb7c61551884a"
server_key_iv: "a290171ba20966ce476142cfdfe6a087"
server_key_version: "encryptionkey@0"
I20260929 02:49:36.764503 9503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:36.769950 9503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:36.770680 9503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:36.770958 9503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:36.787281 9503 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:33395
I20260929 02:49:36.787312 9568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:33395 every 8 connection(s)
I20260929 02:49:36.788606 9503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:36.792344 9569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:36.795419 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 9503
I20260929 02:49:36.795565 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:36.796165 4886 external_mini_cluster.cc:1428] Setting key 0ef20da2ef85b84ac6069dec3f7ba260
I20260929 02:49:36.804198 9569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0: Bootstrap starting.
I20260929 02:49:36.806991 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:36.798030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.4.197.190:33395)
0929 02:49:36.799056 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:36.799068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:36.799109 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:49:36.800631 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:36.800653 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:36.801108 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:36.801463 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:36.805338 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:36.805345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:36.805359 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:36.805380 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:36.805445 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:36.805470 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:36.805650 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:36.805763 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 02:49:36.806012 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":346,"threads_started":1}
I20260929 02:49:36.807701 9569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:36.808897 9569 log.cc:824] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:36.811652 9569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0: No bootstrap required, opened a new log
I20260929 02:49:36.818496 9569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8912ccb28df541e69ad689b93e4145e0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33395 } }
I20260929 02:49:36.818828 9569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:36.818912 9569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8912ccb28df541e69ad689b93e4145e0, State: Initialized, Role: FOLLOWER
I20260929 02:49:36.819415 9569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8912ccb28df541e69ad689b93e4145e0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33395 } }
I20260929 02:49:36.819621 9569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:36.819739 9569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:36.819872 9569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:36.821861 9569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8912ccb28df541e69ad689b93e4145e0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33395 } }
I20260929 02:49:36.822392 9569 leader_election.cc:302] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8912ccb28df541e69ad689b93e4145e0; no voters:
I20260929 02:49:36.823007 9569 leader_election.cc:288] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:36.823097 9574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:36.823700 9574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8912ccb28df541e69ad689b93e4145e0, State: Running, Role: LEADER
I20260929 02:49:36.824218 9574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8912ccb28df541e69ad689b93e4145e0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33395 } }
I20260929 02:49:36.824922 9569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:36.826749 9576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8912ccb28df541e69ad689b93e4145e0. Latest consensus state: current_term: 1 leader_uuid: "8912ccb28df541e69ad689b93e4145e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8912ccb28df541e69ad689b93e4145e0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33395 } } }
I20260929 02:49:36.827351 9576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:36.826777 9575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8912ccb28df541e69ad689b93e4145e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8912ccb28df541e69ad689b93e4145e0" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33395 } } }
I20260929 02:49:36.827692 9575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:36.828121 9579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:36.835795 9579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:36.847513 9579 catalog_manager.cc:1347] Generated new cluster ID: 05ed51affae14e99874225a4baae6590
I20260929 02:49:36.847622 9579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:36.867558 9579 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:36.867981 9579 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:36.877848 9579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0: Generated new TSK 0
I20260929 02:49:36.878676 9579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:49:36.894650 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33395
--builtin_ntp_servers=127.4.197.148:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:49:37.208639 9593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:37.208953 9593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:37.209044 9593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:37.216313 9593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:37.216416 9593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:37.216526 9593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:37.225736 9593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:37.227093 9593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:37.229061 9593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:37.242990 9598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:37.242904 9599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:37.242957 9601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:37.243999 9593 server_base.cc:1034] running on GCE node
I20260929 02:49:37.244583 9593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:37.245586 9593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:37.246795 9593 hybrid_clock.cc:648] HybridClock initialized: now 1790650177246730 us; error 52 us; skew 500 ppm
I20260929 02:49:37.247166 9593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:49:37.249476 9593 webserver.cc:460] Webserver started at http://127.4.197.129:35643/ using document root <none> and password file <none>
I20260929 02:49:37.250175 9593 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:37.250277 9593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:37.250581 9593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:37.253476 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "98d9f1fea40642a3aea88b5ca4ca0f39"
format_stamp: "Formatted at 2026-09-29 02:49:37 on dist-test-slave-t6c1"
server_key: "28d1f81babbbbbdbd2b7de5c9cbc49d5"
server_key_iv: "85bfb34f98253f4a65677dd0d752a759"
server_key_version: "encryptionkey@0"
I20260929 02:49:37.254238 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "98d9f1fea40642a3aea88b5ca4ca0f39"
format_stamp: "Formatted at 2026-09-29 02:49:37 on dist-test-slave-t6c1"
server_key: "28d1f81babbbbbdbd2b7de5c9cbc49d5"
server_key_iv: "85bfb34f98253f4a65677dd0d752a759"
server_key_version: "encryptionkey@0"
I20260929 02:49:37.259245 9593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:49:37.262527 9607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:37.263849 9593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:37.263981 9593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "98d9f1fea40642a3aea88b5ca4ca0f39"
format_stamp: "Formatted at 2026-09-29 02:49:37 on dist-test-slave-t6c1"
server_key: "28d1f81babbbbbdbd2b7de5c9cbc49d5"
server_key_iv: "85bfb34f98253f4a65677dd0d752a759"
server_key_version: "encryptionkey@0"
I20260929 02:49:37.264186 9593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:37.269635 9593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:37.270359 9593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:37.270591 9593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:37.271502 9593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:37.272850 9593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:37.272933 9593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:37.273039 9593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:37.273116 9593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:37.302445 9593 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:42425
I20260929 02:49:37.302495 9719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:42425 every 8 connection(s)
I20260929 02:49:37.303785 9593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:37.311270 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:37.306254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:58615 (local address 127.4.197.190:33395)
0929 02:49:37.306450 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:37.306455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:37.307811 (+ 1356us) server_negotiation.cc:407] Connection header received
0929 02:49:37.309364 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:37.309369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:37.309417 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:37.309498 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:37.310684 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:37.310688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:37.310691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:37.310707 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:37.310741 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:37.310752 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:37.310852 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:37.311037 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 02:49:37.311086 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:49:37.311978 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 9593
I20260929 02:49:37.312134 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:37.312211 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:37.306780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33395 (local address 127.4.197.129:58615)
0929 02:49:37.307663 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:37.307698 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:37.309142 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:37.309675 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:37.309683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:37.309777 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:37.310409 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:49:37.310411 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:37.310448 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:49:37.310529 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:37.310842 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:37.310847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:37.310853 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:49:37.311091 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 02:49:37.311282 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":272,"threads_started":1}
I20260929 02:49:37.312799 4886 external_mini_cluster.cc:1428] Setting key 02fbd231819191f1f89df476b69663ff
I20260929 02:49:37.313664 9720 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33395
I20260929 02:49:37.313969 9720 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:37.314642 9720 heartbeater.cc:511] Master 127.4.197.190:33395 requested a full tablet report, sending...
I20260929 02:49:37.315523 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33395
--builtin_ntp_servers=127.4.197.148:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 02:49:37.316704 9534 ts_manager.cc:194] Registered new tserver with Master: 98d9f1fea40642a3aea88b5ca4ca0f39 (127.4.197.129:42425)
W20260929 02:49:37.625515 9723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:37.625793 9723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:37.625895 9723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:37.633003 9723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:37.633106 9723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:37.633214 9723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:37.642217 9723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:37.643523 9723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:37.645253 9723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:37.657605 9728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:37.657742 9729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:37.657878 9731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:37.658556 9723 server_base.cc:1034] running on GCE node
I20260929 02:49:37.659173 9723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:37.660056 9723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:37.661238 9723 hybrid_clock.cc:648] HybridClock initialized: now 1790650177661176 us; error 54 us; skew 500 ppm
I20260929 02:49:37.661542 9723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:49:37.663789 9723 webserver.cc:460] Webserver started at http://127.4.197.130:44745/ using document root <none> and password file <none>
I20260929 02:49:37.664482 9723 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:37.664574 9723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:37.664884 9723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:37.667827 9723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "ef9bdd7de1e547bea70e439001837020"
format_stamp: "Formatted at 2026-09-29 02:49:37 on dist-test-slave-t6c1"
server_key: "b09e9577d5a7b7fdd648cdfaeed653ad"
server_key_iv: "d115a9415481f5bfbb9c882f708df499"
server_key_version: "encryptionkey@0"
I20260929 02:49:37.668686 9723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "ef9bdd7de1e547bea70e439001837020"
format_stamp: "Formatted at 2026-09-29 02:49:37 on dist-test-slave-t6c1"
server_key: "b09e9577d5a7b7fdd648cdfaeed653ad"
server_key_iv: "d115a9415481f5bfbb9c882f708df499"
server_key_version: "encryptionkey@0"
I20260929 02:49:37.673947 9723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:49:37.677762 9737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:37.679255 9723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:49:37.679404 9723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "ef9bdd7de1e547bea70e439001837020"
format_stamp: "Formatted at 2026-09-29 02:49:37 on dist-test-slave-t6c1"
server_key: "b09e9577d5a7b7fdd648cdfaeed653ad"
server_key_iv: "d115a9415481f5bfbb9c882f708df499"
server_key_version: "encryptionkey@0"
I20260929 02:49:37.679628 9723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:37.685102 9723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:37.685851 9723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:37.686092 9723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:37.686973 9723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:37.688282 9723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:37.688364 9723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:37.688467 9723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:37.688529 9723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:37.716431 9723 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:45329
I20260929 02:49:37.716478 9849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:45329 every 8 connection(s)
I20260929 02:49:37.717600 9723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:37.722265 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 9723
I20260929 02:49:37.722395 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:37.722826 4886 external_mini_cluster.cc:1428] Setting key 9ab4bf5dff8d9dd7fc62e7d0c4fc7987
I20260929 02:49:37.725211 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:37.719933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:48727 (local address 127.4.197.190:33395)
0929 02:49:37.720091 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:37.720096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:37.721413 (+ 1317us) server_negotiation.cc:407] Connection header received
0929 02:49:37.723136 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:37.723142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:37.723210 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:37.723301 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:37.724498 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:37.724504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:37.724509 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:37.724528 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:37.724567 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:37.724581 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:37.724695 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:37.724858 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 02:49:37.724920 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:49:37.725404 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33395
--builtin_ntp_servers=127.4.197.148:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 02:49:37.726266 9852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:37.720418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33395 (local address 127.4.197.130:48727)
0929 02:49:37.721265 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:37.721303 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:37.722824 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:37.723524 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:37.723532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:37.723622 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:37.724219 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:49:37.724222 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:37.724258 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:49:37.724339 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:37.724668 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:37.724678 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:37.724685 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:49:37.724934 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 02:49:37.725184 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":243,"threads_started":1}
I20260929 02:49:37.727607 9850 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33395
I20260929 02:49:37.727860 9850 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:37.728392 9850 heartbeater.cc:511] Master 127.4.197.190:33395 requested a full tablet report, sending...
I20260929 02:49:37.729676 9534 ts_manager.cc:194] Registered new tserver with Master: ef9bdd7de1e547bea70e439001837020 (127.4.197.130:45329)
W20260929 02:49:38.041671 9853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:38.041955 9853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:38.042085 9853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:38.049391 9853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:38.049499 9853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:38.049654 9853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:38.059314 9853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:38.060576 9853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:38.062306 9853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:38.074994 9861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:38.074908 9859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:38.075027 9858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:38.076082 9853 server_base.cc:1034] running on GCE node
I20260929 02:49:38.076710 9853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:38.077675 9853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:38.078877 9853 hybrid_clock.cc:648] HybridClock initialized: now 1790650178078824 us; error 42 us; skew 500 ppm
I20260929 02:49:38.079228 9853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:49:38.081454 9853 webserver.cc:460] Webserver started at http://127.4.197.131:34035/ using document root <none> and password file <none>
I20260929 02:49:38.082130 9853 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:38.082243 9853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:38.082564 9853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:38.085508 9853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "aef2005d0c784c10824da43784d35c4d"
format_stamp: "Formatted at 2026-09-29 02:49:38 on dist-test-slave-t6c1"
server_key: "0f0e6da9cc58bc1373692d4d4cd2ab07"
server_key_iv: "31c047acda7b9de32568d2d7b806dad8"
server_key_version: "encryptionkey@0"
I20260929 02:49:38.086288 9853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "aef2005d0c784c10824da43784d35c4d"
format_stamp: "Formatted at 2026-09-29 02:49:38 on dist-test-slave-t6c1"
server_key: "0f0e6da9cc58bc1373692d4d4cd2ab07"
server_key_iv: "31c047acda7b9de32568d2d7b806dad8"
server_key_version: "encryptionkey@0"
I20260929 02:49:38.091396 9853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:49:38.094758 9867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:38.096078 9853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:38.096215 9853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "aef2005d0c784c10824da43784d35c4d"
format_stamp: "Formatted at 2026-09-29 02:49:38 on dist-test-slave-t6c1"
server_key: "0f0e6da9cc58bc1373692d4d4cd2ab07"
server_key_iv: "31c047acda7b9de32568d2d7b806dad8"
server_key_version: "encryptionkey@0"
I20260929 02:49:38.096423 9853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:38.101600 9853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:38.102298 9853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:38.102532 9853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:38.103408 9853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:38.104638 9853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:38.104719 9853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:38.104820 9853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:38.104897 9853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:38.131366 9853 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:43225
I20260929 02:49:38.131417 9979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:43225 every 8 connection(s)
I20260929 02:49:38.132647 9853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:38.136991 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 9853
I20260929 02:49:38.137140 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:38.137563 4886 external_mini_cluster.cc:1428] Setting key 25244783e67296395943076766f8812d
I20260929 02:49:38.140904 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.135104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:48791 (local address 127.4.197.190:33395)
0929 02:49:38.135308 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.135314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.136835 (+ 1521us) server_negotiation.cc:407] Connection header received
0929 02:49:38.138616 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.138621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.138681 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.138811 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.140271 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.140277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.140281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.140300 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.140338 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.140353 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.140471 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.140664 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.140713 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:49:38.141912 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.135722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33395 (local address 127.4.197.131:48791)
0929 02:49:38.136676 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.136719 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:38.138380 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:38.139085 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:38.139098 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:38.139233 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:38.139958 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:49:38.139962 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:38.140005 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:49:38.140106 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:38.140465 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:38.140473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:38.140479 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:49:38.140734 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 02:49:38.140980 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":274,"threads_started":1}
I20260929 02:49:38.143075 9980 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33395
I20260929 02:49:38.143337 9980 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:38.143895 9980 heartbeater.cc:511] Master 127.4.197.190:33395 requested a full tablet report, sending...
I20260929 02:49:38.145119 9534 ts_manager.cc:194] Registered new tserver with Master: aef2005d0c784c10824da43784d35c4d (127.4.197.131:43225)
I20260929 02:49:38.152809 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:38.159314 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.157135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.4.197.190:33395)
0929 02:49:38.157319 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.157325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.157745 (+ 420us) server_negotiation.cc:407] Connection header received
0929 02:49:38.157878 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.157882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.157932 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.158009 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.158765 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.158769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.158772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.158786 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.158817 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.158828 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.158943 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.159006 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.159052 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:49:38.163128 9534 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:38.169952 9534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:38.187528 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.181760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42425 (local address 127.0.0.1:50924)
0929 02:49:38.182783 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.182828 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:38.183005 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:38.184243 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:38.184253 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:38.184383 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:38.185245 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:49:38.185249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:38.185283 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:49:38.185389 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:38.186689 (+ 1300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:38.186701 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:38.186711 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:49:38.186879 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 02:49:38.187005 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":808,"threads_started":1}
I20260929 02:49:38.188575 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.181950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.4.197.129:42425)
0929 02:49:38.183474 (+ 1524us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.183485 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.183518 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:49:38.183643 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.183652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.183922 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.184911 (+ 989us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.186437 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.186446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.186463 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.186487 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.186541 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.186560 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.188042 (+ 1482us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.188120 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.188224 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1372,"thread_start_us":1250,"threads_started":1}
I20260929 02:49:38.192047 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.182374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.4.197.130:45329)
0929 02:49:38.189757 (+ 7383us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.189768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.189800 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:49:38.189868 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.189878 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.190130 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.190289 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.191303 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.191311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.191325 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.191346 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.191409 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.191428 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.191587 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.191652 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.191739 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7226,"thread_start_us":7103,"threads_started":1}
I20260929 02:49:38.191984 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.181909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:45329 (local address 127.0.0.1:54098)
0929 02:49:38.187767 (+ 5858us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.187787 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:38.187868 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:38.190323 (+ 2455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:38.190329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:38.190377 (+ 48us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:38.191027 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:49:38.191031 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:38.191051 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:49:38.191101 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:38.191529 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:38.191534 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:38.191539 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:49:38.191625 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 02:49:38.191674 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5753,"thread_start_us":6992,"threads_started":1}
I20260929 02:49:38.193451 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.184065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:43225 (local address 127.0.0.1:43594)
0929 02:49:38.189198 (+ 5133us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.189217 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:38.189309 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:38.191655 (+ 2346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:38.191660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:38.191681 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:38.192481 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:49:38.192485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:38.192499 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:49:38.192539 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:38.192908 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:38.192912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:38.192917 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:49:38.193029 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 02:49:38.193073 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4995,"thread_start_us":8195,"threads_started":1}
I20260929 02:49:38.193451 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.184687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.4.197.131:43225)
0929 02:49:38.190144 (+ 5457us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.190152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.190186 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:49:38.190249 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.190255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.190453 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.190589 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.192680 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.192688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.192699 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.192717 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.192771 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.192790 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.192963 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.193022 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.193100 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5307,"thread_start_us":5141,"threads_started":1}
I20260929 02:49:38.196152 9785 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3873943d614b5a9816df75250078b6 (DEFAULT_TABLE table=test-table [id=18dddf9ae20e45408470b698aa2a97d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:38.196152 9655 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3873943d614b5a9816df75250078b6 (DEFAULT_TABLE table=test-table [id=18dddf9ae20e45408470b698aa2a97d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:38.196499 9915 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3873943d614b5a9816df75250078b6 (DEFAULT_TABLE table=test-table [id=18dddf9ae20e45408470b698aa2a97d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:38.197507 9655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3873943d614b5a9816df75250078b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:38.197507 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3873943d614b5a9816df75250078b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:38.197575 9915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3873943d614b5a9816df75250078b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:38.208945 9998 tablet_bootstrap.cc:492] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: Bootstrap starting.
I20260929 02:49:38.210951 10000 tablet_bootstrap.cc:492] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: Bootstrap starting.
I20260929 02:49:38.212052 9999 tablet_bootstrap.cc:492] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39: Bootstrap starting.
I20260929 02:49:38.212504 9998 tablet_bootstrap.cc:654] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:38.213737 9998 log.cc:824] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:38.214723 10000 tablet_bootstrap.cc:654] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:38.215046 9999 tablet_bootstrap.cc:654] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:38.215700 10000 log.cc:824] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:38.216099 9999 log.cc:824] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:38.216769 9998 tablet_bootstrap.cc:492] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: No bootstrap required, opened a new log
I20260929 02:49:38.217029 9998 ts_tablet_manager.cc:1397] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:49:38.218266 9999 tablet_bootstrap.cc:492] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39: No bootstrap required, opened a new log
I20260929 02:49:38.218381 10000 tablet_bootstrap.cc:492] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: No bootstrap required, opened a new log
I20260929 02:49:38.218483 9999 ts_tablet_manager.cc:1397] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 02:49:38.218611 10000 ts_tablet_manager.cc:1397] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260929 02:49:38.226786 9998 raft_consensus.cc:348] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 } } peers { permanent_uuid: "ef9bdd7de1e547bea70e439001837020" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45329 } } peers { permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 } }
I20260929 02:49:38.227254 9998 raft_consensus.cc:374] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:38.227355 9998 raft_consensus.cc:729] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef2005d0c784c10824da43784d35c4d, State: Initialized, Role: FOLLOWER
I20260929 02:49:38.227980 9998 consensus_queue.cc:260] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 } } peers { permanent_uuid: "ef9bdd7de1e547bea70e439001837020" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45329 } } peers { permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 } }
I20260929 02:49:38.228183 9999 raft_consensus.cc:348] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 } } peers { permanent_uuid: "ef9bdd7de1e547bea70e439001837020" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45329 } } peers { permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 } }
I20260929 02:49:38.228571 9999 raft_consensus.cc:374] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:38.228667 9999 raft_consensus.cc:729] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d9f1fea40642a3aea88b5ca4ca0f39, State: Initialized, Role: FOLLOWER
I20260929 02:49:38.229313 9999 consensus_queue.cc:260] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 } } peers { permanent_uuid: "ef9bdd7de1e547bea70e439001837020" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45329 } } peers { permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 } }
I20260929 02:49:38.229503 10000 raft_consensus.cc:348] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 } } peers { permanent_uuid: "ef9bdd7de1e547bea70e439001837020" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45329 } } peers { permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 } }
I20260929 02:49:38.229916 10000 raft_consensus.cc:374] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:38.230006 10000 raft_consensus.cc:729] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9bdd7de1e547bea70e439001837020, State: Initialized, Role: FOLLOWER
I20260929 02:49:38.230588 9980 heartbeater.cc:503] Master 127.4.197.190:33395 was elected leader, sending a full tablet report...
I20260929 02:49:38.231159 9998 ts_tablet_manager.cc:1428] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260929 02:49:38.231835 9720 heartbeater.cc:503] Master 127.4.197.190:33395 was elected leader, sending a full tablet report...
I20260929 02:49:38.232170 10000 consensus_queue.cc:260] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 } } peers { permanent_uuid: "ef9bdd7de1e547bea70e439001837020" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45329 } } peers { permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 } }
I20260929 02:49:38.232388 9999 ts_tablet_manager.cc:1428] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 02:49:38.233251 9850 heartbeater.cc:503] Master 127.4.197.190:33395 was elected leader, sending a full tablet report...
I20260929 02:49:38.234324 10000 ts_tablet_manager.cc:1428] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
W20260929 02:49:38.305781 9721 tablet.cc:2369] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:49:38.384173 9981 tablet.cc:2369] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:38.442286 10006 raft_consensus.cc:482] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:38.442610 10006 raft_consensus.cc:504] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 } } peers { permanent_uuid: "ef9bdd7de1e547bea70e439001837020" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45329 } } peers { permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 } }
I20260929 02:49:38.444235 10006 leader_election.cc:288] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98d9f1fea40642a3aea88b5ca4ca0f39 (127.4.197.129:42425), aef2005d0c784c10824da43784d35c4d (127.4.197.131:43225)
I20260929 02:49:38.447242 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.444746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:42151 (local address 127.4.197.129:42425)
0929 02:49:38.444960 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.444967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.445200 (+ 233us) server_negotiation.cc:407] Connection header received
0929 02:49:38.445448 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.445454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.445516 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.445582 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.446652 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.446656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.446660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.446675 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.446703 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.446713 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.446833 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.446901 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.446976 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:49:38.447520 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.445295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:34413 (local address 127.4.197.131:43225)
0929 02:49:38.445487 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.445492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.445882 (+ 390us) server_negotiation.cc:407] Connection header received
0929 02:49:38.445953 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.445958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.446014 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.446108 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.446909 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.446935 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.446941 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.446960 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.446993 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.447006 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.447111 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.447195 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.447256 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:49:38.447650 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.445115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:43225 (local address 127.4.197.130:34413)
0929 02:49:38.445756 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.445774 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:38.445873 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:38.446150 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:38.446153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:38.446174 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:38.446730 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:49:38.446732 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:38.446745 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:49:38.446777 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:38.447091 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:38.447095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:38.447100 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:49:38.447240 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 02:49:38.447286 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":384,"threads_started":1}
I20260929 02:49:38.447775 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.444478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42425 (local address 127.4.197.130:42151)
0929 02:49:38.445094 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.445115 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:38.445283 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:38.445754 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:38.445757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:38.445776 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:38.446464 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:49:38.446466 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:38.446480 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:49:38.446516 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:38.446798 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:38.446803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:38.446807 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 02:49:38.446906 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:49:38.446972 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":283,"threads_started":1}
I20260929 02:49:38.448717 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3873943d614b5a9816df75250078b6" candidate_uuid: "ef9bdd7de1e547bea70e439001837020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aef2005d0c784c10824da43784d35c4d" is_pre_election: true
I20260929 02:49:38.448717 9675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3873943d614b5a9816df75250078b6" candidate_uuid: "ef9bdd7de1e547bea70e439001837020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" is_pre_election: true
I20260929 02:49:38.449260 9675 raft_consensus.cc:2454] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef9bdd7de1e547bea70e439001837020 in term 0.
I20260929 02:49:38.449262 9935 raft_consensus.cc:2454] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef9bdd7de1e547bea70e439001837020 in term 0.
I20260929 02:49:38.449887 9740 leader_election.cc:302] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98d9f1fea40642a3aea88b5ca4ca0f39, ef9bdd7de1e547bea70e439001837020; no voters:
I20260929 02:49:38.450214 10006 raft_consensus.cc:2781] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:38.450354 10006 raft_consensus.cc:482] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:38.450456 10006 raft_consensus.cc:3037] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:38.452579 10006 raft_consensus.cc:504] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 } } peers { permanent_uuid: "ef9bdd7de1e547bea70e439001837020" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45329 } } peers { permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 } }
I20260929 02:49:38.453109 10006 leader_election.cc:288] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [CANDIDATE]: Term 1 election: Requested vote from peers 98d9f1fea40642a3aea88b5ca4ca0f39 (127.4.197.129:42425), aef2005d0c784c10824da43784d35c4d (127.4.197.131:43225)
I20260929 02:49:38.453500 9675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3873943d614b5a9816df75250078b6" candidate_uuid: "ef9bdd7de1e547bea70e439001837020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39"
I20260929 02:49:38.453686 9675 raft_consensus.cc:3037] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:38.453657 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3873943d614b5a9816df75250078b6" candidate_uuid: "ef9bdd7de1e547bea70e439001837020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aef2005d0c784c10824da43784d35c4d"
I20260929 02:49:38.453908 9935 raft_consensus.cc:3037] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:38.456084 9675 raft_consensus.cc:2454] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef9bdd7de1e547bea70e439001837020 in term 1.
I20260929 02:49:38.456262 9935 raft_consensus.cc:2454] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef9bdd7de1e547bea70e439001837020 in term 1.
I20260929 02:49:38.456504 9740 leader_election.cc:302] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98d9f1fea40642a3aea88b5ca4ca0f39, ef9bdd7de1e547bea70e439001837020; no voters:
I20260929 02:49:38.456817 10006 raft_consensus.cc:2781] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:38.457262 10006 raft_consensus.cc:686] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 1 LEADER]: Becoming Leader. State: Replica: ef9bdd7de1e547bea70e439001837020, State: Running, Role: LEADER
I20260929 02:49:38.457736 10006 consensus_queue.cc:237] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 } } peers { permanent_uuid: "ef9bdd7de1e547bea70e439001837020" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45329 } } peers { permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 } }
I20260929 02:49:38.462855 9532 catalog_manager.cc:5495] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef9bdd7de1e547bea70e439001837020 (127.4.197.130). New cstate: current_term: 1 leader_uuid: "ef9bdd7de1e547bea70e439001837020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef9bdd7de1e547bea70e439001837020" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 } health_report { overall_health: UNKNOWN } } }
W20260929 02:49:38.469300 9851 tablet.cc:2369] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:38.491191 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.489267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.4.197.130:45329)
0929 02:49:38.489447 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.489451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.489580 (+ 129us) server_negotiation.cc:407] Connection header received
0929 02:49:38.489766 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.489769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.489853 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.489937 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.490713 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.490717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.490721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.490737 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.490770 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.490781 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.490877 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.490931 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.490992 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:49:38.499837 9935 raft_consensus.cc:1261] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d [term 1 FOLLOWER]: Refusing update from remote peer ef9bdd7de1e547bea70e439001837020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:38.499811 9675 raft_consensus.cc:1261] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39 [term 1 FOLLOWER]: Refusing update from remote peer ef9bdd7de1e547bea70e439001837020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:38.500857 10009 consensus_queue.cc:1035] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef2005d0c784c10824da43784d35c4d" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:38.501282 10006 consensus_queue.cc:1035] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d9f1fea40642a3aea88b5ca4ca0f39" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:38.527000 9532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:38.527262 9532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:38.530489 9532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8912ccb28df541e69ad689b93e4145e0: Sending DeleteTablet for 3 replicas of tablet 8c3873943d614b5a9816df75250078b6
I20260929 02:49:38.531701 9785 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3873943d614b5a9816df75250078b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:38 UTC) from {username='slave'} at 127.0.0.1:54098
I20260929 02:49:38.531946 9655 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3873943d614b5a9816df75250078b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:38 UTC) from {username='slave'} at 127.0.0.1:50924
I20260929 02:49:38.532253 9915 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3873943d614b5a9816df75250078b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:38 UTC) from {username='slave'} at 127.0.0.1:43594
I20260929 02:49:38.533318 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 9593
I20260929 02:49:38.545037 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.534660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.4.197.129:42425)
0929 02:49:38.534836 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.534842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.540884 (+ 6042us) server_negotiation.cc:407] Connection header received
0929 02:49:38.541754 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.541761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.541885 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.541962 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.544037 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.544044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.544050 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.544072 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.544115 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.544131 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.544229 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.544631 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.544699 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:49:38.546137 9715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:38.561619 10024 tablet_replica.cc:330] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: stopping tablet replica
I20260929 02:49:38.561995 10022 tablet_replica.cc:330] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: stopping tablet replica
I20260929 02:49:38.562160 10024 raft_consensus.cc:2229] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:38.562603 10022 raft_consensus.cc:2229] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:38.562743 10024 raft_consensus.cc:2258] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:38.563416 10022 raft_consensus.cc:2258] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:38.565725 10022 ts_tablet_manager.cc:1905] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:38.569118 10024 ts_tablet_manager.cc:1905] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:38.573618 10022 ts_tablet_manager.cc:1918] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:38.573732 10024 ts_tablet_manager.cc:1918] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:38.573781 10022 log.cc:1196] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/8c3873943d614b5a9816df75250078b6
I20260929 02:49:38.573879 10024 log.cc:1196] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/8c3873943d614b5a9816df75250078b6
I20260929 02:49:38.574270 10024 ts_tablet_manager.cc:1939] T 8c3873943d614b5a9816df75250078b6 P aef2005d0c784c10824da43784d35c4d: Deleting consensus metadata
I20260929 02:49:38.574342 10022 ts_tablet_manager.cc:1939] T 8c3873943d614b5a9816df75250078b6 P ef9bdd7de1e547bea70e439001837020: Deleting consensus metadata
I20260929 02:49:38.576006 9520 catalog_manager.cc:4841] TS aef2005d0c784c10824da43784d35c4d (127.4.197.131:43225): tablet 8c3873943d614b5a9816df75250078b6 (table test-table [id=18dddf9ae20e45408470b698aa2a97d2]) successfully deleted
I20260929 02:49:38.576068 9519 catalog_manager.cc:4841] TS ef9bdd7de1e547bea70e439001837020 (127.4.197.130:45329): tablet 8c3873943d614b5a9816df75250078b6 (table test-table [id=18dddf9ae20e45408470b698aa2a97d2]) successfully deleted
I20260929 02:49:38.660391 10023 tablet_replica.cc:330] T 8c3873943d614b5a9816df75250078b6 P 98d9f1fea40642a3aea88b5ca4ca0f39: stopping tablet replica
I20260929 02:49:38.660773 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 9593
I20260929 02:49:38.673988 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 9723
W20260929 02:49:38.674023 9520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.197.129:42425 (error 108)
I20260929 02:49:38.675835 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.675428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42425 (local address 127.0.0.1:50948)
0929 02:49:38.675594 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.675711 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:49:38.676091 9520 catalog_manager.cc:4584] TS 98d9f1fea40642a3aea88b5ca4ca0f39 (127.4.197.129:42425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c3873943d614b5a9816df75250078b6: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42425: connect: Connection refused (error 111)
I20260929 02:49:38.677174 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.675084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.4.197.130:45329)
0929 02:49:38.675240 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.675244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.675259 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:49:38.675339 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.675346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.675403 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.675504 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.676530 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.676534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.676537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.676553 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.676582 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.676594 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.676695 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.676846 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.676938 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:49:38.677875 9845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:38.729773 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.729133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42425 (local address 127.0.0.1:50958)
0929 02:49:38.729389 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.729533 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:49:38.730984 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.730557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42425 (local address 127.0.0.1:50974)
0929 02:49:38.730708 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.730801 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 02:49:38.791735 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.791253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42425 (local address 127.0.0.1:50976)
0929 02:49:38.791427 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.791554 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 02:49:38.792984 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.792639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42425 (local address 127.0.0.1:50986)
0929 02:49:38.792773 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.792848 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 02:49:38.795285 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 9723
I20260929 02:49:38.808897 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 9853
I20260929 02:49:38.811460 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.809523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.4.197.131:43225)
0929 02:49:38.809697 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:38.809702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:38.809721 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:49:38.809931 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:38.809936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:38.810023 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:38.810103 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:38.810842 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:38.810848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:38.810853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:38.810870 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:38.810903 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:38.810941 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:38.811051 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:38.811134 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 02:49:38.811202 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:49:38.811997 9975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:38.873839 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.873306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42425 (local address 127.0.0.1:50988)
0929 02:49:38.873551 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.873660 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 02:49:38.875039 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.874560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42425 (local address 127.0.0.1:50994)
0929 02:49:38.874781 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.874865 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 02:49:38.897547 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 9853
I20260929 02:49:38.909795 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 9503
I20260929 02:49:38.910892 9564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:39.013763 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 9503
2026-09-29T02:49:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2671 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 02:49:39.049417 4886 mini_oidc.cc:152] Starting JWKS server
I20260929 02:49:39.051949 4886 webserver.cc:460] Webserver started at https://127.0.0.1:42763/ using document root <none> and password file <none>
I20260929 02:49:39.052227 4886 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:49:39.052804 4886 webserver.cc:460] Webserver started at http://127.0.0.1:33703/ using document root <none> and password file <none>
2026-09-29T02:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:39Z Disabled control of system clock
I20260929 02:49:39.075815 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:43723
--webserver_interface=127.4.197.190
--webserver_port=0
--builtin_ntp_servers=127.4.197.148:33529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:43723
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42763/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/testchainca.cert with env {}
W20260929 02:49:39.386524 10036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:39.386847 10036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:39.386979 10036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:39.394343 10036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:39.394456 10036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:39.394500 10036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:39.394552 10036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:39.394595 10036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:49:39.394654 10036 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:49:39.394690 10036 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42763/jwks.json
I20260929 02:49:39.404536 10036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:33529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:43723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:43723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42763/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:39.405748 10036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:39.407021 10036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:39.417613 10044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:39.417755 10041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:39.417711 10042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:39.418594 10036 server_base.cc:1034] running on GCE node
I20260929 02:49:39.419286 10036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:39.420531 10036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:39.421708 10036 hybrid_clock.cc:648] HybridClock initialized: now 1790650179421665 us; error 39 us; skew 500 ppm
I20260929 02:49:39.422009 10036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:49:39.424081 10036 webserver.cc:460] Webserver started at http://127.4.197.190:44581/ using document root <none> and password file <none>
I20260929 02:49:39.424706 10036 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:39.424796 10036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:39.425076 10036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:39.427908 10036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "aaf562fbe31f42c79292d623e1acbf76"
format_stamp: "Formatted at 2026-09-29 02:49:39 on dist-test-slave-t6c1"
server_key: "ebff93590af83abb1e7eaf6251f294ab"
server_key_iv: "bd7685b6bf672e4779526f9dac961e2b"
server_key_version: "encryptionkey@0"
I20260929 02:49:39.428658 10036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "aaf562fbe31f42c79292d623e1acbf76"
format_stamp: "Formatted at 2026-09-29 02:49:39 on dist-test-slave-t6c1"
server_key: "ebff93590af83abb1e7eaf6251f294ab"
server_key_iv: "bd7685b6bf672e4779526f9dac961e2b"
server_key_version: "encryptionkey@0"
I20260929 02:49:39.433568 10036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:49:39.437112 10050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:39.438472 10036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:49:39.438623 10036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "aaf562fbe31f42c79292d623e1acbf76"
format_stamp: "Formatted at 2026-09-29 02:49:39 on dist-test-slave-t6c1"
server_key: "ebff93590af83abb1e7eaf6251f294ab"
server_key_iv: "bd7685b6bf672e4779526f9dac961e2b"
server_key_version: "encryptionkey@0"
I20260929 02:49:39.438813 10036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:39.453488 10036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:39.454351 10036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:39.454641 10036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:39.471153 10036 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:43723
I20260929 02:49:39.471202 10101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:43723 every 8 connection(s)
I20260929 02:49:39.472496 10036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:39.476183 10102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:39.481864 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 10036
I20260929 02:49:39.482101 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:39.482882 4886 external_mini_cluster.cc:1428] Setting key c1d5b97320d21091345485487bd8be81
I20260929 02:49:39.487782 10102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76: Bootstrap starting.
I20260929 02:49:39.490986 10102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:39.491950 10102 log.cc:824] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:39.495005 10102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76: No bootstrap required, opened a new log
I20260929 02:49:39.496335 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:39.484913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.4.197.190:43723)
0929 02:49:39.486019 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:39.486030 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:39.486070 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:49:39.487773 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:39.487796 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:39.488323 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:39.488707 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:39.489313 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:39.490587 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:39.491975 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:39.492886 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:39.493072 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:39.494134 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:39.494141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:39.494155 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:39.494661 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:39.494732 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:39.494751 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:39.494835 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:39.495089 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 02:49:39.495416 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":376,"threads_started":1}
I20260929 02:49:39.500128 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:49:39.504168 10102 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf562fbe31f42c79292d623e1acbf76" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43723 } }
I20260929 02:49:39.504488 10102 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:39.504560 10102 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf562fbe31f42c79292d623e1acbf76, State: Initialized, Role: FOLLOWER
I20260929 02:49:39.505045 10102 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf562fbe31f42c79292d623e1acbf76" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43723 } }
I20260929 02:49:39.505244 10102 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:39.505339 10102 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:39.505451 10102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:39.508105 10102 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf562fbe31f42c79292d623e1acbf76" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43723 } }
I20260929 02:49:39.508698 10102 leader_election.cc:302] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaf562fbe31f42c79292d623e1acbf76; no voters:
I20260929 02:49:39.509416 10102 leader_election.cc:288] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:39.509526 10107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:39.510176 10107 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [term 1 LEADER]: Becoming Leader. State: Replica: aaf562fbe31f42c79292d623e1acbf76, State: Running, Role: LEADER
I20260929 02:49:39.510637 10107 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf562fbe31f42c79292d623e1acbf76" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43723 } }
I20260929 02:49:39.511363 10102 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:39.513134 10108 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaf562fbe31f42c79292d623e1acbf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf562fbe31f42c79292d623e1acbf76" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43723 } } }
I20260929 02:49:39.513499 10108 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:39.514146 10109 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaf562fbe31f42c79292d623e1acbf76. Latest consensus state: current_term: 1 leader_uuid: "aaf562fbe31f42c79292d623e1acbf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf562fbe31f42c79292d623e1acbf76" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43723 } } }
I20260929 02:49:39.514454 10109 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:39.515849 10113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:39.519752 10113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:39.528110 10113 catalog_manager.cc:1347] Generated new cluster ID: 04b8f4147eed4465b340cbfca0fe8af7
I20260929 02:49:39.528451 10113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:39.537278 10113 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:39.538516 10113 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:39.547737 10113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaf562fbe31f42c79292d623e1acbf76: Generated new TSK 0
I20260929 02:49:39.548640 10113 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 02:49:39.589231 10132 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 02:49:39.589385 10132 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:49:39.590600 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:39.569534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59842 (local address 127.4.197.190:43723)
0929 02:49:39.569727 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:39.569734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:39.570089 (+ 355us) server_negotiation.cc:407] Connection header received
0929 02:49:39.570417 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:39.570421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:39.570497 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:39.570620 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:49:39.571208 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:39.571977 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:39.573344 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:39.573794 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:39.573915 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:39.574492 (+ 577us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:49:39.589918 (+ 15426us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 02:49:39.590313 (+ 395us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 02:49:39.590725 10105 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 02:49:39.592027 10131 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:49:39.569366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:43723 (local address 127.0.0.1:59842)
0929 02:49:39.569982 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:39.570001 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:39.570179 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:39.570681 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:39.570686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:39.570699 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:49:39.571062 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:39.571072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:39.572115 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:39.572121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:39.573191 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:39.573203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:39.573954 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:39.573960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:39.574113 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:39.574179 (+ 66us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:49:39.590219 (+ 16040us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 02:49:39.590503 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.197.190:43723: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":371,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.197.190:43723: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 02:49:39.593597 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 10036
I20260929 02:49:39.595093 10097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:39.687153 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 10036
2026-09-29T02:49:39Z chronyd exiting
I20260929 02:49:39.733428 4886 test_util.cc:175] -----------------------------------------------
I20260929 02:49:39.733610 4886 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650137986910-4886-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (688 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 02:49:39.738440 4886 mini_oidc.cc:152] Starting JWKS server
I20260929 02:49:39.740305 4886 webserver.cc:460] Webserver started at https://127.0.0.1:43031/ using document root <none> and password file <none>
I20260929 02:49:39.740490 4886 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:49:39.741062 4886 webserver.cc:460] Webserver started at http://127.0.0.1:36099/ using document root <none> and password file <none>
2026-09-29T02:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:39Z Disabled control of system clock
I20260929 02:49:39.764632 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:41367
--webserver_interface=127.4.197.190
--webserver_port=0
--builtin_ntp_servers=127.4.197.148:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:41367
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43031/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/testchainca.cert with env {}
W20260929 02:49:40.082374 10139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:40.082684 10139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:40.082787 10139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:40.089970 10139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:40.090082 10139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:40.090118 10139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:40.090174 10139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:40.090232 10139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:49:40.090312 10139 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:49:40.090355 10139 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43031/jwks.json
I20260929 02:49:40.100414 10139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:41367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:41367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43031/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:40.101542 10139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:40.102728 10139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:40.113116 10145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:40.113453 10147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:40.113492 10144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:40.113885 10139 server_base.cc:1034] running on GCE node
I20260929 02:49:40.114559 10139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:40.115782 10139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:40.116968 10139 hybrid_clock.cc:648] HybridClock initialized: now 1790650180116924 us; error 38 us; skew 500 ppm
I20260929 02:49:40.117270 10139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:49:40.119212 10139 webserver.cc:460] Webserver started at http://127.4.197.190:46281/ using document root <none> and password file <none>
I20260929 02:49:40.119792 10139 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:40.119887 10139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:40.120162 10139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:40.122725 10139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "df9e9e95a2de446c84dd2bf3ef133035"
format_stamp: "Formatted at 2026-09-29 02:49:40 on dist-test-slave-t6c1"
server_key: "7858e6a944831b32359bce92a9f9e8ab"
server_key_iv: "04deb4af935e2660cfa73c0cd9ddca25"
server_key_version: "encryptionkey@0"
I20260929 02:49:40.123484 10139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "df9e9e95a2de446c84dd2bf3ef133035"
format_stamp: "Formatted at 2026-09-29 02:49:40 on dist-test-slave-t6c1"
server_key: "7858e6a944831b32359bce92a9f9e8ab"
server_key_iv: "04deb4af935e2660cfa73c0cd9ddca25"
server_key_version: "encryptionkey@0"
I20260929 02:49:40.128127 10139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:49:40.131318 10153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:40.132619 10139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:49:40.132769 10139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "df9e9e95a2de446c84dd2bf3ef133035"
format_stamp: "Formatted at 2026-09-29 02:49:40 on dist-test-slave-t6c1"
server_key: "7858e6a944831b32359bce92a9f9e8ab"
server_key_iv: "04deb4af935e2660cfa73c0cd9ddca25"
server_key_version: "encryptionkey@0"
I20260929 02:49:40.132965 10139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:40.146615 10139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:40.147511 10139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:40.147791 10139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:40.164184 10139 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:41367
I20260929 02:49:40.164197 10204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:41367 every 8 connection(s)
I20260929 02:49:40.165498 10139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:40.169097 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:40.171896 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 10139
I20260929 02:49:40.172096 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:40.172878 4886 external_mini_cluster.cc:1428] Setting key 5272cc836ea931181fb1e4b883d3c281
I20260929 02:49:40.181368 10205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035: Bootstrap starting.
I20260929 02:49:40.184922 10205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:40.186081 10205 log.cc:824] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:40.186546 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:40.174566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.4.197.190:41367)
0929 02:49:40.175633 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:40.175648 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:40.175696 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:49:40.177673 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:40.177700 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:40.178190 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:40.178529 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:40.179107 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.180387 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.182139 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.182833 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.183007 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:40.184151 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:40.184158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:40.184175 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:40.184700 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:40.184788 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:40.184809 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:40.184892 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:40.185160 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 02:49:40.185509 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":352,"threads_started":1}
I20260929 02:49:40.189142 10205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035: No bootstrap required, opened a new log
I20260929 02:49:40.190297 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:49:40.196399 10205 raft_consensus.cc:348] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e9e95a2de446c84dd2bf3ef133035" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41367 } }
I20260929 02:49:40.196744 10205 raft_consensus.cc:374] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:40.196832 10205 raft_consensus.cc:729] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9e9e95a2de446c84dd2bf3ef133035, State: Initialized, Role: FOLLOWER
I20260929 02:49:40.197362 10205 consensus_queue.cc:260] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e9e95a2de446c84dd2bf3ef133035" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41367 } }
I20260929 02:49:40.197566 10205 raft_consensus.cc:388] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:40.197677 10205 raft_consensus.cc:482] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:40.197826 10205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:40.200057 10205 raft_consensus.cc:504] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e9e95a2de446c84dd2bf3ef133035" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41367 } }
I20260929 02:49:40.200498 10205 leader_election.cc:302] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df9e9e95a2de446c84dd2bf3ef133035; no voters:
I20260929 02:49:40.201031 10205 leader_election.cc:288] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:40.201161 10210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:40.201745 10210 raft_consensus.cc:686] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [term 1 LEADER]: Becoming Leader. State: Replica: df9e9e95a2de446c84dd2bf3ef133035, State: Running, Role: LEADER
I20260929 02:49:40.202247 10210 consensus_queue.cc:237] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e9e95a2de446c84dd2bf3ef133035" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41367 } }
I20260929 02:49:40.202852 10205 sys_catalog.cc:564] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:40.204845 10212 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df9e9e95a2de446c84dd2bf3ef133035. Latest consensus state: current_term: 1 leader_uuid: "df9e9e95a2de446c84dd2bf3ef133035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e9e95a2de446c84dd2bf3ef133035" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41367 } } }
I20260929 02:49:40.205240 10212 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:40.205399 10211 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df9e9e95a2de446c84dd2bf3ef133035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e9e95a2de446c84dd2bf3ef133035" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41367 } } }
I20260929 02:49:40.206887 10219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:40.206936 10211 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:40.211719 10219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:40.219750 10219 catalog_manager.cc:1347] Generated new cluster ID: ce9bf6e980644220afe69a4e778c1e47
I20260929 02:49:40.219851 10219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:40.229379 10219 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:40.231175 10219 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:40.243753 10219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df9e9e95a2de446c84dd2bf3ef133035: Generated new TSK 0
I20260929 02:49:40.244900 10219 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 02:49:40.281173 10235 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 02:49:40.281291 10235 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:49:40.282646 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:40.262287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.4.197.190:41367)
0929 02:49:40.262505 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:40.262511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:40.262977 (+ 466us) server_negotiation.cc:407] Connection header received
0929 02:49:40.263251 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:40.263258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:40.263367 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:40.263553 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:49:40.263971 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.265108 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.266463 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.267179 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.267330 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:40.267894 (+ 564us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:49:40.281928 (+ 14034us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 02:49:40.282336 (+ 408us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 02:49:40.282688 10234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:49:40.262104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:41367 (local address 127.0.0.1:56668)
0929 02:49:40.262847 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:40.262868 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:40.263043 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:40.263554 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:40.263557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:40.263566 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:49:40.263812 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:40.263819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.265238 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.265242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:40.266317 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:40.266328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.267381 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.267386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:40.267524 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:40.267558 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:49:40.282178 (+ 14620us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 02:49:40.282391 (+ 213us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.197.190:41367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":446,"threads_started":1}
W20260929 02:49:40.282780 10208 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 02:49:40.284108 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 10139
I20260929 02:49:40.285216 10200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:40.377010 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 10139
2026-09-29T02:49:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (691 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 02:49:40.428419 4886 mini_oidc.cc:152] Starting JWKS server
I20260929 02:49:40.430205 4886 webserver.cc:460] Webserver started at https://127.0.0.1:37443/ using document root <none> and password file <none>
I20260929 02:49:40.430421 4886 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:49:40.431032 4886 webserver.cc:460] Webserver started at http://127.0.0.1:37823/ using document root <none> and password file <none>
2026-09-29T02:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:40Z Disabled control of system clock
I20260929 02:49:40.457247 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34969
--webserver_interface=127.4.197.190
--webserver_port=0
--builtin_ntp_servers=127.4.197.148:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:34969
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37443/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 02:49:40.764943 10242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:40.765264 10242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:40.765388 10242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:40.772535 10242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:40.772675 10242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:40.772750 10242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:40.772828 10242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:40.772886 10242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:49:40.772969 10242 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:49:40.773021 10242 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37443/jwks.json
I20260929 02:49:40.782611 10242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:34969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37443/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:40.783943 10242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:40.785640 10242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:40.796726 10247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:40.796751 10248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:40.796658 10250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:40.797528 10242 server_base.cc:1034] running on GCE node
I20260929 02:49:40.798270 10242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:40.799634 10242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:40.800832 10242 hybrid_clock.cc:648] HybridClock initialized: now 1790650180800781 us; error 42 us; skew 500 ppm
I20260929 02:49:40.801142 10242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:49:40.803259 10242 webserver.cc:460] Webserver started at http://127.4.197.190:36711/ using document root <none> and password file <none>
I20260929 02:49:40.803892 10242 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:40.803987 10242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:40.804281 10242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:40.807003 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "bc606a5d5cbc43b78739733f85b5f58d"
format_stamp: "Formatted at 2026-09-29 02:49:40 on dist-test-slave-t6c1"
server_key: "6bc885388dec1efed9e6827187fa85ea"
server_key_iv: "ab5edb5735fea81a0164f493629cbe62"
server_key_version: "encryptionkey@0"
I20260929 02:49:40.807735 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "bc606a5d5cbc43b78739733f85b5f58d"
format_stamp: "Formatted at 2026-09-29 02:49:40 on dist-test-slave-t6c1"
server_key: "6bc885388dec1efed9e6827187fa85ea"
server_key_iv: "ab5edb5735fea81a0164f493629cbe62"
server_key_version: "encryptionkey@0"
I20260929 02:49:40.812330 10242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:49:40.815356 10256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:40.816529 10242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 02:49:40.816673 10242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "bc606a5d5cbc43b78739733f85b5f58d"
format_stamp: "Formatted at 2026-09-29 02:49:40 on dist-test-slave-t6c1"
server_key: "6bc885388dec1efed9e6827187fa85ea"
server_key_iv: "ab5edb5735fea81a0164f493629cbe62"
server_key_version: "encryptionkey@0"
I20260929 02:49:40.816846 10242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:40.841429 10242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:40.842289 10242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:40.842595 10242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:40.859453 10242 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:34969
I20260929 02:49:40.859472 10307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:34969 every 8 connection(s)
I20260929 02:49:40.860770 10242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:40.864331 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 10242
I20260929 02:49:40.864543 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:40.864696 10308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:40.865327 4886 external_mini_cluster.cc:1428] Setting key 41e2af12a7c634d4f3cca85badd0afc0
I20260929 02:49:40.877643 10308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d: Bootstrap starting.
I20260929 02:49:40.878439 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:40.867135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.4.197.190:34969)
0929 02:49:40.868157 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:40.868169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:40.868211 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:49:40.869933 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:40.869956 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:40.870463 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:40.870877 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:40.871493 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.872792 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.874066 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.874750 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.874966 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:40.876126 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:40.876133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:40.876148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:40.876705 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:40.876787 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:40.876809 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:40.876914 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:40.877132 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 02:49:40.877440 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":338,"threads_started":1}
I20260929 02:49:40.881260 10308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:40.882356 10308 log.cc:824] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:40.882445 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:49:40.885138 10308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d: No bootstrap required, opened a new log
I20260929 02:49:40.892746 10308 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc606a5d5cbc43b78739733f85b5f58d" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34969 } }
I20260929 02:49:40.893153 10308 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:40.893244 10308 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc606a5d5cbc43b78739733f85b5f58d, State: Initialized, Role: FOLLOWER
I20260929 02:49:40.894003 10308 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc606a5d5cbc43b78739733f85b5f58d" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34969 } }
I20260929 02:49:40.894255 10308 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:40.894394 10308 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:40.894549 10308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:40.896835 10308 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc606a5d5cbc43b78739733f85b5f58d" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34969 } }
I20260929 02:49:40.897342 10308 leader_election.cc:302] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc606a5d5cbc43b78739733f85b5f58d; no voters:
I20260929 02:49:40.897893 10308 leader_election.cc:288] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:40.898022 10313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:40.899115 10313 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [term 1 LEADER]: Becoming Leader. State: Replica: bc606a5d5cbc43b78739733f85b5f58d, State: Running, Role: LEADER
I20260929 02:49:40.899605 10313 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc606a5d5cbc43b78739733f85b5f58d" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34969 } }
I20260929 02:49:40.900048 10308 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:40.902035 10314 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc606a5d5cbc43b78739733f85b5f58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc606a5d5cbc43b78739733f85b5f58d" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34969 } } }
I20260929 02:49:40.902369 10314 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:40.902886 10315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc606a5d5cbc43b78739733f85b5f58d. Latest consensus state: current_term: 1 leader_uuid: "bc606a5d5cbc43b78739733f85b5f58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc606a5d5cbc43b78739733f85b5f58d" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34969 } } }
I20260929 02:49:40.903306 10315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:40.904626 10319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:40.909925 10319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:40.918401 10319 catalog_manager.cc:1347] Generated new cluster ID: 8b51c33e4fd14595bf62aeea25c21c7c
I20260929 02:49:40.918491 10319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:40.935422 10319 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:40.936734 10319 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:40.944761 10319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc606a5d5cbc43b78739733f85b5f58d: Generated new TSK 0
I20260929 02:49:40.945829 10319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:49:40.978107 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:40.970099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.4.197.190:34969)
0929 02:49:40.970315 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:40.970322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:40.970576 (+ 254us) server_negotiation.cc:407] Connection header received
0929 02:49:40.970777 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:40.970783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:40.970872 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:40.971067 (+ 195us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:49:40.971479 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.972574 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.973856 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.974546 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.974662 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:40.975153 (+ 491us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:49:40.977482 (+ 2329us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 02:49:40.977803 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33144: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 02:49:40.978236 10311 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33144: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 02:49:40.979897 10337 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:49:40.969916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34969 (local address 127.0.0.1:33144)
0929 02:49:40.970479 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:40.970495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:40.970614 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:40.971075 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:40.971078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:40.971088 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:49:40.971342 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:40.971350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.972697 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.972701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:40.973728 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:40.973738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:40.974701 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:40.974705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:40.974845 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:40.974875 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:49:40.977784 (+ 2909us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 02:49:40.979480 (+ 1696us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.197.190:34969: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":1415,"thread_start_us":284,"threads_started":1}
I20260929 02:49:40.981861 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 10242
I20260929 02:49:40.982982 10303 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 02:49:40.989492 10338 webserver.cc:533] Webserver: sslize failed:
E20260929 02:49:40.989580 10338 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:49:41.072440 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 10242
2026-09-29T02:49:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (694 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T02:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:41Z Disabled control of system clock
I20260929 02:49:41.145859 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45679
--webserver_interface=127.4.197.190
--webserver_port=0
--builtin_ntp_servers=127.4.197.148:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:45679
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/testchainca.cert with env {}
W20260929 02:49:41.457547 10343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:41.458017 10343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:41.458168 10343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:41.465472 10343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:41.465590 10343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:41.465626 10343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:41.465680 10343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:41.465729 10343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:49:41.465787 10343 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:49:41.465831 10343 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 02:49:41.475798 10343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:45679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:41.477092 10343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:41.478339 10343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:41.490089 10348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:41.490104 10351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:41.490097 10349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:41.490994 10343 server_base.cc:1034] running on GCE node
I20260929 02:49:41.491772 10343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:41.493242 10343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:41.494465 10343 hybrid_clock.cc:648] HybridClock initialized: now 1790650181494402 us; error 60 us; skew 500 ppm
I20260929 02:49:41.494823 10343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:49:41.497097 10343 webserver.cc:460] Webserver started at http://127.4.197.190:34569/ using document root <none> and password file <none>
I20260929 02:49:41.497798 10343 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:41.497913 10343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:41.498211 10343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:41.501566 10343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "bfa535c5a90d41a5a14269183aaff359"
format_stamp: "Formatted at 2026-09-29 02:49:41 on dist-test-slave-t6c1"
server_key: "6d9ce8b236c22d2dd49f24527a00e4ca"
server_key_iv: "641e369918fac89e740bee5865e116dd"
server_key_version: "encryptionkey@0"
I20260929 02:49:41.502413 10343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "bfa535c5a90d41a5a14269183aaff359"
format_stamp: "Formatted at 2026-09-29 02:49:41 on dist-test-slave-t6c1"
server_key: "6d9ce8b236c22d2dd49f24527a00e4ca"
server_key_iv: "641e369918fac89e740bee5865e116dd"
server_key_version: "encryptionkey@0"
I20260929 02:49:41.507846 10343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:49:41.511281 10357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:41.512650 10343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:49:41.512910 10343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "bfa535c5a90d41a5a14269183aaff359"
format_stamp: "Formatted at 2026-09-29 02:49:41 on dist-test-slave-t6c1"
server_key: "6d9ce8b236c22d2dd49f24527a00e4ca"
server_key_iv: "641e369918fac89e740bee5865e116dd"
server_key_version: "encryptionkey@0"
I20260929 02:49:41.513149 10343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:41.542166 10343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:41.543099 10343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:41.543380 10343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:41.560474 10343 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:45679
I20260929 02:49:41.560509 10408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:45679 every 8 connection(s)
I20260929 02:49:41.561791 10343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:41.562896 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 10343
I20260929 02:49:41.563107 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:41.563902 4886 external_mini_cluster.cc:1428] Setting key 47b6c2981ce80707feb50e78502acee0
I20260929 02:49:41.566504 10409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:41.576967 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:41.565895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.4.197.190:45679)
0929 02:49:41.566869 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:41.566887 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:41.566953 (+ 66us) server_negotiation.cc:407] Connection header received
0929 02:49:41.568586 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:41.568608 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:41.569106 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:41.569491 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:41.570042 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:41.571423 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:41.572660 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:41.573309 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:41.573479 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:41.574662 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:41.574671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:41.574689 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:49:41.575311 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:41.575404 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:49:41.575428 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:41.575527 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:41.575741 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 02:49:41.576044 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":297,"threads_started":1}
I20260929 02:49:41.577148 10409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359: Bootstrap starting.
I20260929 02:49:41.580698 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:49:41.580714 10409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:41.581769 10409 log.cc:824] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:41.584479 10409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359: No bootstrap required, opened a new log
I20260929 02:49:41.591490 10409 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa535c5a90d41a5a14269183aaff359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45679 } }
I20260929 02:49:41.591873 10409 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:41.591960 10409 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa535c5a90d41a5a14269183aaff359, State: Initialized, Role: FOLLOWER
I20260929 02:49:41.592550 10409 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa535c5a90d41a5a14269183aaff359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45679 } }
I20260929 02:49:41.592808 10409 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:41.592939 10409 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:41.593102 10409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:41.595270 10409 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa535c5a90d41a5a14269183aaff359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45679 } }
I20260929 02:49:41.595793 10409 leader_election.cc:302] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfa535c5a90d41a5a14269183aaff359; no voters:
I20260929 02:49:41.596388 10409 leader_election.cc:288] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:41.596521 10414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:41.597132 10414 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [term 1 LEADER]: Becoming Leader. State: Replica: bfa535c5a90d41a5a14269183aaff359, State: Running, Role: LEADER
I20260929 02:49:41.597715 10414 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa535c5a90d41a5a14269183aaff359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45679 } }
I20260929 02:49:41.598479 10409 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:41.600692 10416 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfa535c5a90d41a5a14269183aaff359. Latest consensus state: current_term: 1 leader_uuid: "bfa535c5a90d41a5a14269183aaff359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa535c5a90d41a5a14269183aaff359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45679 } } }
I20260929 02:49:41.601051 10416 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:41.601221 10415 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfa535c5a90d41a5a14269183aaff359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa535c5a90d41a5a14269183aaff359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45679 } } }
I20260929 02:49:41.601589 10415 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:41.603144 10422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:41.608140 10422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:41.616510 10422 catalog_manager.cc:1347] Generated new cluster ID: e3067433bfe74bfe842352b7150fc674
I20260929 02:49:41.616606 10422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:41.628770 10422 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:41.630070 10422 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:41.639425 10422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfa535c5a90d41a5a14269183aaff359: Generated new TSK 0
I20260929 02:49:41.640528 10422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:49:41.752816 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:41.661893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.4.197.190:45679)
0929 02:49:41.662077 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:41.662083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:41.662409 (+ 326us) server_negotiation.cc:407] Connection header received
0929 02:49:41.662615 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:41.662621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:41.662705 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:41.662857 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:49:41.663333 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:41.664466 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:41.665823 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:41.666510 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:41.666643 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:41.667084 (+ 441us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:49:41.752065 (+ 84981us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 02:49:41.752507 (+ 442us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45214: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 02:49:41.752923 10438 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:49:41.661739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45679 (local address 127.0.0.1:45214)
0929 02:49:41.662300 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:41.662315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:41.662453 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:41.662865 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:41.662868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:41.662878 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:49:41.663169 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:41.663178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:41.664625 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:41.664629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:41.665686 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:41.665697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:41.666650 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:41.666653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:41.666785 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:41.666817 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:49:41.752381 (+ 85564us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 02:49:41.752616 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.197.190:45679: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":281,"threads_started":1}
W20260929 02:49:41.752981 10412 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45214: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260929 02:49:41.754066 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 10343
I20260929 02:49:41.755228 10404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:41.855510 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 10343
2026-09-29T02:49:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (763 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (392 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (398 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:42 dist-test-slave-t6c1 krb5kdc[10453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:42 dist-test-slave-t6c1 krb5kdc[10453](info): set up 2 sockets
Sep 29 02:49:42 dist-test-slave-t6c1 krb5kdc[10453](info): commencing operation
krb5kdc: starting...
W20260929 02:49:43.647540 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:49:43 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:49:43.843080 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:38737
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:38737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:44.151634 10466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:44.151976 10466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:44.152103 10466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:44.159454 10466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:44.159582 10466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:44.159631 10466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:44.159709 10466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:44.159765 10466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:44.169869 10466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:38737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:38737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:44.171314 10466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:44.172650 10466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:44.184098 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:44.184038 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:44.184027 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:44.185192 10466 server_base.cc:1034] running on GCE node
I20260929 02:49:44.185962 10466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:44.187563 10466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:44.188769 10466 hybrid_clock.cc:648] HybridClock initialized: now 1790650184188711 us; error 50 us; skew 500 ppm
I20260929 02:49:44.189092 10466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:44 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650184, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:44.192049 10466 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:49:44.193384 10466 webserver.cc:460] Webserver started at http://127.4.197.190:38927/ using document root <none> and password file <none>
I20260929 02:49:44.194017 10466 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:44.194128 10466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:44.194423 10466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:44.197455 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3"
format_stamp: "Formatted at 2026-09-29 02:49:44 on dist-test-slave-t6c1"
server_key: "96bcc12e09835b3cbe6b94e7af93945e"
server_key_iv: "943ddcb21965f6d5fa2bd7c8ee2c9c1d"
server_key_version: "encryptionkey@0"
I20260929 02:49:44.198199 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3"
format_stamp: "Formatted at 2026-09-29 02:49:44 on dist-test-slave-t6c1"
server_key: "96bcc12e09835b3cbe6b94e7af93945e"
server_key_iv: "943ddcb21965f6d5fa2bd7c8ee2c9c1d"
server_key_version: "encryptionkey@0"
I20260929 02:49:44.203033 10466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 02:49:44.206151 10481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:44.207408 10466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:49:44.207551 10466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3"
format_stamp: "Formatted at 2026-09-29 02:49:44 on dist-test-slave-t6c1"
server_key: "96bcc12e09835b3cbe6b94e7af93945e"
server_key_iv: "943ddcb21965f6d5fa2bd7c8ee2c9c1d"
server_key_version: "encryptionkey@0"
I20260929 02:49:44.207706 10466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:44.235246 10466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:44.239773 10466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:44.240054 10466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:44.256917 10466 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:38737
I20260929 02:49:44.256925 10532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:38737 every 8 connection(s)
I20260929 02:49:44.258240 10466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:44.261829 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 10466
I20260929 02:49:44.261977 10533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:44.262033 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:44.262646 4886 external_mini_cluster.cc:1428] Setting key bc96eb0423a971169441becd85b9be74
I20260929 02:49:44.275086 10533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3: Bootstrap starting.
Sep 29 02:49:44 dist-test-slave-t6c1 krb5kdc[10453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:44.278707 10533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:44.279843 10533 log.cc:824] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:44.282797 10533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3: No bootstrap required, opened a new log
I20260929 02:49:44.285624 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:44.264555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.4.197.190:38737)
0929 02:49:44.265307 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:44.265319 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:44.265366 (+ 47us) server_negotiation.cc:407] Connection header received
0929 02:49:44.267711 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:44.267734 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:44.268142 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:44.268480 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:44.270100 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:44.271429 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:44.273117 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:44.273762 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:44.273884 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:44.278452 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:44.278460 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:44.278476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:44.278516 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:44.281560 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:44.282260 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:44.282266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:44.282273 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:44.282469 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:44.282848 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:44.282853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:44.282857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:44.283430 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:44.283720 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:44.284156 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 02:49:44.284594 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":348,"threads_started":1}
I20260929 02:49:44.291118 10533 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38737 } }
I20260929 02:49:44.291476 10533 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:44.291567 10533 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdfae7d60ff4aaca7ede630a7aba2c3, State: Initialized, Role: FOLLOWER
I20260929 02:49:44.292089 10533 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38737 } }
I20260929 02:49:44.292310 10533 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:44.292430 10533 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:44.292577 10533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:44.294899 10533 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38737 } }
I20260929 02:49:44.295415 10533 leader_election.cc:302] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbdfae7d60ff4aaca7ede630a7aba2c3; no voters:
I20260929 02:49:44.295903 10533 leader_election.cc:288] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:44.296020 10538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:44.296586 10538 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [term 1 LEADER]: Becoming Leader. State: Replica: bbdfae7d60ff4aaca7ede630a7aba2c3, State: Running, Role: LEADER
I20260929 02:49:44.297070 10538 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38737 } }
I20260929 02:49:44.297739 10533 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:44.299597 10540 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbdfae7d60ff4aaca7ede630a7aba2c3. Latest consensus state: current_term: 1 leader_uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38737 } } }
I20260929 02:49:44.299606 10539 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdfae7d60ff4aaca7ede630a7aba2c3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38737 } } }
I20260929 02:49:44.300001 10539 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:44.300001 10540 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:44.302376 10544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:44.307240 10544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:44.315536 10544 catalog_manager.cc:1347] Generated new cluster ID: 2f3d1ced0c8045e69068d6b9004bc7b2
I20260929 02:49:44.315680 10544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:44.323879 10544 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:44.325163 10544 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:44.332603 10544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbdfae7d60ff4aaca7ede630a7aba2c3: Generated new TSK 0
I20260929 02:49:44.333715 10544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:49:44.388670 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38737
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:44.724650 10559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:44.724967 10559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:44.725100 10559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:44.732517 10559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:44.732656 10559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:44.732808 10559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:44.742638 10559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:44.743994 10559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:44.745381 10559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:44.758337 10564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:44.758363 10567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:44.758579 10565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:44.759517 10559 server_base.cc:1034] running on GCE node
I20260929 02:49:44.760166 10559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:44.761096 10559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:44.762296 10559 hybrid_clock.cc:648] HybridClock initialized: now 1790650184762224 us; error 60 us; skew 500 ppm
I20260929 02:49:44.762634 10559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:44 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650184, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:44.765877 10559 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:49:44.767387 10559 webserver.cc:460] Webserver started at http://127.4.197.129:46119/ using document root <none> and password file <none>
I20260929 02:49:44.768057 10559 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:44.768158 10559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:44.768428 10559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:44.771418 10559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "46cfcb17168a4ee99a722fe5d991742a"
format_stamp: "Formatted at 2026-09-29 02:49:44 on dist-test-slave-t6c1"
server_key: "84801523a84a57f2fa11fa3ef8b7f991"
server_key_iv: "00e2cafebeafdab9b1ffe7e114f325b1"
server_key_version: "encryptionkey@0"
I20260929 02:49:44.772159 10559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "46cfcb17168a4ee99a722fe5d991742a"
format_stamp: "Formatted at 2026-09-29 02:49:44 on dist-test-slave-t6c1"
server_key: "84801523a84a57f2fa11fa3ef8b7f991"
server_key_iv: "00e2cafebeafdab9b1ffe7e114f325b1"
server_key_version: "encryptionkey@0"
I20260929 02:49:44.777303 10559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:49:44.781018 10574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:44.782508 10559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:49:44.782651 10559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "46cfcb17168a4ee99a722fe5d991742a"
format_stamp: "Formatted at 2026-09-29 02:49:44 on dist-test-slave-t6c1"
server_key: "84801523a84a57f2fa11fa3ef8b7f991"
server_key_iv: "00e2cafebeafdab9b1ffe7e114f325b1"
server_key_version: "encryptionkey@0"
I20260929 02:49:44.782831 10559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:44.796041 10559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:44.799952 10559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:44.800187 10559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:44.801069 10559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:44.802404 10559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:44.802474 10559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:44.802563 10559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:44.802608 10559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:44.831074 10559 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:37747
I20260929 02:49:44.831107 10686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:37747 every 8 connection(s)
I20260929 02:49:44.832324 10559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:44.840169 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 10559
I20260929 02:49:44.840353 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:44.840957 4886 external_mini_cluster.cc:1428] Setting key aeaa3f0982607dd8d03bd014d29dd3bb
Sep 29 02:49:44 dist-test-slave-t6c1 krb5kdc[10453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650184, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:44.852782 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:44.834769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:49341 (local address 127.4.197.190:38737)
0929 02:49:44.835195 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:44.835199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:44.836309 (+ 1110us) server_negotiation.cc:407] Connection header received
0929 02:49:44.838112 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:44.838118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:44.838207 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:44.838300 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:44.839988 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:44.840778 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:44.842772 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:44.843269 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:44.843399 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:44.847053 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:44.847058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:44.847062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:44.847095 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:44.849741 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:44.850480 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:44.850484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:44.850486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:44.850605 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:44.851063 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:44.851066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:44.851068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:44.851337 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:44.851488 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:44.852191 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 02:49:44.852428 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":236,"threads_started":1}
I20260929 02:49:44.853773 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:44.835401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38737 (local address 127.4.197.129:49341)
0929 02:49:44.836146 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:44.836187 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:44.837831 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:44.838553 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:44.838561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:44.839124 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:44.839767 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:44.839794 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:44.840989 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:44.840995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:44.842618 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:44.842635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:44.843508 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:44.843515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:44.843737 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:44.844424 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:44.844445 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:44.846769 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:44.849896 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:44.849904 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:44.849918 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:44.850340 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:44.850709 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:44.850713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:44.850715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:44.850951 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:44.851468 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:44.851476 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:44.851824 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 02:49:44.852068 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 02:49:44.852615 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":311,"threads_started":1}
I20260929 02:49:44.855336 10687 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38737
I20260929 02:49:44.855667 10687 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:44.856427 10687 heartbeater.cc:511] Master 127.4.197.190:38737 requested a full tablet report, sending...
I20260929 02:49:44.858697 10498 ts_manager.cc:194] Registered new tserver with Master: 46cfcb17168a4ee99a722fe5d991742a (127.4.197.129:37747)
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:49:44.860384 10498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:49341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:49:44.886325 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38737
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:45.201610 10693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:45.201954 10693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:45.202099 10693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:45.209550 10693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:45.209700 10693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:45.209861 10693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:45.219780 10693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:45.221025 10693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:45.222411 10693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:45.235961 10699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:45.236106 10698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:45.236003 10701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:45.237102 10693 server_base.cc:1034] running on GCE node
I20260929 02:49:45.237715 10693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:45.238651 10693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:45.239846 10693 hybrid_clock.cc:648] HybridClock initialized: now 1790650185239782 us; error 55 us; skew 500 ppm
I20260929 02:49:45.240149 10693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:45.243209 10693 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:49:45.244747 10693 webserver.cc:460] Webserver started at http://127.4.197.130:39773/ using document root <none> and password file <none>
I20260929 02:49:45.245402 10693 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:45.245517 10693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:45.245810 10693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:45.248669 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "2391a97037894d3282d3a0de9346fcd0"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-t6c1"
server_key: "916305adadc856146e93f31228a0ffbe"
server_key_iv: "2d101707471bb7cc0b4069027625a938"
server_key_version: "encryptionkey@0"
I20260929 02:49:45.249402 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "2391a97037894d3282d3a0de9346fcd0"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-t6c1"
server_key: "916305adadc856146e93f31228a0ffbe"
server_key_iv: "2d101707471bb7cc0b4069027625a938"
server_key_version: "encryptionkey@0"
I20260929 02:49:45.254426 10693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:49:45.258333 10708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:45.259966 10693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:45.260133 10693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "2391a97037894d3282d3a0de9346fcd0"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-t6c1"
server_key: "916305adadc856146e93f31228a0ffbe"
server_key_iv: "2d101707471bb7cc0b4069027625a938"
server_key_version: "encryptionkey@0"
I20260929 02:49:45.260356 10693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:45.275399 10693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:45.279091 10693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:45.279381 10693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:45.280241 10693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:45.281457 10693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:45.281525 10693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:45.281634 10693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:45.281694 10693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:45.310673 10693 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:42731
I20260929 02:49:45.310688 10820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:42731 every 8 connection(s)
I20260929 02:49:45.312098 10693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:45.313488 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 10693
I20260929 02:49:45.313668 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:45.314172 4886 external_mini_cluster.cc:1428] Setting key bb492f8787e27c3e44b9d938028ad594
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
I20260929 02:49:45.332108 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:45.314776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:53343 (local address 127.4.197.190:38737)
0929 02:49:45.314980 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:45.314987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:45.316561 (+ 1574us) server_negotiation.cc:407] Connection header received
0929 02:49:45.318435 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:45.318440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:45.318515 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:45.318609 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:45.320105 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.321042 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.323025 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.323426 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.323536 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:45.326850 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:45.326855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:45.326858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:45.326891 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:45.329353 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.329990 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.329993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.329996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.330115 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.330502 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.330506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.330508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.330753 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:45.330884 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:45.331557 (+ 673us) server_negotiation.cc:299] Negotiation successful
0929 02:49:45.331766 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:49:45.333048 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:45.315525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38737 (local address 127.4.197.130:53343)
0929 02:49:45.316386 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:45.316437 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:45.318168 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:45.318844 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:45.318853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:45.319385 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:45.319950 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:45.319968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.321174 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.321178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:45.322883 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:45.322895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.323633 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.323640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:45.323861 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:45.324494 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:45.324516 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:45.326629 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:45.329471 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:45.329480 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:45.329495 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:45.329874 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:45.330209 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:45.330214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:45.330218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:45.330395 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:45.330864 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:45.330871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:45.331205 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 02:49:45.331440 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 02:49:45.331934 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":340,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
I20260929 02:49:45.334497 10821 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38737
I20260929 02:49:45.334761 10821 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:45.335481 10821 heartbeater.cc:511] Master 127.4.197.190:38737 requested a full tablet report, sending...
I20260929 02:49:45.336789 10498 ts_manager.cc:194] Registered new tserver with Master: 2391a97037894d3282d3a0de9346fcd0 (127.4.197.130:42731)
I20260929 02:49:45.337525 10498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:53343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:49:45.356012 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38737
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:45.669869 10826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:45.670145 10826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:45.670264 10826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:45.677289 10826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:45.677397 10826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:45.677513 10826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:45.686947 10826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:45.688200 10826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:45.689481 10826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:45.701766 10832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:45.701720 10834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:45.701695 10831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:45.702457 10826 server_base.cc:1034] running on GCE node
I20260929 02:49:45.703084 10826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:45.703966 10826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:45.705152 10826 hybrid_clock.cc:648] HybridClock initialized: now 1790650185705098 us; error 50 us; skew 500 ppm
I20260929 02:49:45.705433 10826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:45.708369 10826 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:49:45.709622 10826 webserver.cc:460] Webserver started at http://127.4.197.131:39793/ using document root <none> and password file <none>
I20260929 02:49:45.710235 10826 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:45.710351 10826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:45.710611 10826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:45.713421 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "e2654b0013c34d278320e4b1cb23bfc9"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-t6c1"
server_key: "151f1ac91c5c8c65b2b4cd133ba4f0d2"
server_key_iv: "642c2e89d7f71a50ca6a610f32ffa209"
server_key_version: "encryptionkey@0"
I20260929 02:49:45.714107 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "e2654b0013c34d278320e4b1cb23bfc9"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-t6c1"
server_key: "151f1ac91c5c8c65b2b4cd133ba4f0d2"
server_key_iv: "642c2e89d7f71a50ca6a610f32ffa209"
server_key_version: "encryptionkey@0"
I20260929 02:49:45.718698 10826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260929 02:49:45.721711 10841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:45.722894 10826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:45.723075 10826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "e2654b0013c34d278320e4b1cb23bfc9"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-t6c1"
server_key: "151f1ac91c5c8c65b2b4cd133ba4f0d2"
server_key_iv: "642c2e89d7f71a50ca6a610f32ffa209"
server_key_version: "encryptionkey@0"
I20260929 02:49:45.723239 10826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:45.759147 10826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:45.763392 10826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:45.763669 10826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:45.764729 10826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:45.766113 10826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:45.766196 10826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:45.766325 10826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:45.766386 10826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:45.795814 10826 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:45359
I20260929 02:49:45.795857 10953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:45359 every 8 connection(s)
I20260929 02:49:45.797019 10826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:45.804504 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 10826
I20260929 02:49:45.804656 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:45.805059 4886 external_mini_cluster.cc:1428] Setting key 3f3530e33676a64f989ee739118edaf8
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:49:45.815241 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:45.799471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:36359 (local address 127.4.197.190:38737)
0929 02:49:45.799659 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:45.799665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:45.800846 (+ 1181us) server_negotiation.cc:407] Connection header received
0929 02:49:45.802588 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:45.802592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:45.802658 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:45.802742 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:45.804126 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.805015 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.806852 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.807315 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.807422 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:45.810370 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:45.810375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:45.810378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:45.810406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:45.812634 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.813236 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.813241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.813244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.813350 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.813707 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.813712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.813715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.813973 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:45.814095 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:45.814694 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 02:49:45.814898 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:49:45.816088 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:45.800013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38737 (local address 127.4.197.131:36359)
0929 02:49:45.800692 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:45.800730 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:45.802296 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:45.802961 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:45.802971 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:45.803449 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:45.803998 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:45.804015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.805160 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.805164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:45.806721 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:45.806733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.807517 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.807523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:45.807741 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:45.808277 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:45.808301 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:45.810192 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:45.812748 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:45.812756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:45.812771 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:45.813137 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:45.813441 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:45.813446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:45.813450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:45.813614 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:45.814082 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:45.814089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:45.814378 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 02:49:45.814606 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 02:49:45.815092 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":265,"threads_started":1}
I20260929 02:49:45.817210 10954 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38737
I20260929 02:49:45.817471 10954 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:45.818200 10954 heartbeater.cc:511] Master 127.4.197.190:38737 requested a full tablet report, sending...
I20260929 02:49:45.819569 10497 ts_manager.cc:194] Registered new tserver with Master: e2654b0013c34d278320e4b1cb23bfc9 (127.4.197.131:45359)
I20260929 02:49:45.820360 10497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:36359
I20260929 02:49:45.831979 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/test-admin.keytab.
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:45.857223 4886 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 02:49:45.861967 10961 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:49:45.859703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.4.197.129:37747)
0929 02:49:45.860317 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:45.860323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:45.860350 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:49:45.860409 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:45.860416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:45.860627 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:45.860763 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:45.861612 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58666: BlockingRecv error: recv got EOF from 127.0.0.1:58666 (error 108)
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":322,"threads_started":1}
W20260929 02:49:45.862053 4886 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5cc' of type FILE
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:45.863328 10687 heartbeater.cc:503] Master 127.4.197.190:38737 was elected leader, sending a full tablet report...
I20260929 02:49:45.863533 4886 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:45.864997 4886 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:45.878906 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:45.866520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.4.197.129:37747)
0929 02:49:45.866686 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:45.866691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:45.866809 (+ 118us) server_negotiation.cc:407] Connection header received
0929 02:49:45.866983 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:45.866988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:45.867050 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:45.867125 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:45.868088 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.869021 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.870205 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.870770 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.870892 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:45.874028 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:45.874038 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:45.874050 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:45.874078 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:45.876522 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.877087 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.877094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.877096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.877206 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.877534 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.877538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.877542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.877866 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:45.878115 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:45.878408 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 02:49:45.878647 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 02:49:45.884202 10961 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:49:45.882629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.4.197.129:37747)
0929 02:49:45.882801 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:45.882807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:45.883049 (+ 242us) server_negotiation.cc:407] Connection header received
0929 02:49:45.883203 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:45.883208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:45.883282 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:45.883361 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:45.883989 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58684: BlockingRecv error: recv got EOF from 127.0.0.1:58684 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 02:49:45.884379 4886 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5cc' of type FILE
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:45.885859 4886 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:45.887189 4886 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:45.900048 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:45.888441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.4.197.129:37747)
0929 02:49:45.888634 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:45.888640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:45.888863 (+ 223us) server_negotiation.cc:407] Connection header received
0929 02:49:45.888991 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:45.888995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:45.889050 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:45.889124 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:45.890053 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.890735 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.891912 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.892302 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.892401 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:45.895459 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:45.895464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:45.895467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:45.895511 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:45.897914 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.898497 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.898501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.898503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.898619 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.898906 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.898909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.898911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.899200 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:45.899388 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:45.899608 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 02:49:45.899757 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 02:49:45.903896 10961 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:49:45.902361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.4.197.129:37747)
0929 02:49:45.902554 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:45.902559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:45.902746 (+ 187us) server_negotiation.cc:407] Connection header received
0929 02:49:45.902887 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:45.902890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:45.903006 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:45.903075 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:45.903667 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58700: BlockingRecv error: recv got EOF from 127.0.0.1:58700 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 02:49:45.904013 4886 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650137986910-4886-0/krb5kdc/krb5cc' of type FILE
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:45.905337 4886 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:45.906457 4886 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:49:45 dist-test-slave-t6c1 krb5kdc[10453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:49:45.918910 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:45.907597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.4.197.129:37747)
0929 02:49:45.907773 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:45.907778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:45.907971 (+ 193us) server_negotiation.cc:407] Connection header received
0929 02:49:45.908073 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:45.908076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:45.908124 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:45.908192 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:45.908970 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.909635 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.910723 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.911218 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.911304 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:45.914289 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:45.914293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:45.914297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:45.914327 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:45.916909 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.917439 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.917443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.917445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.917572 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.917836 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.917839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.917841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.918086 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:45.918294 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:45.918457 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 02:49:45.918604 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:49:45.920377 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 10559
I20260929 02:49:45.931264 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:45.920978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.4.197.129:37747)
0929 02:49:45.921132 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:45.921138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:45.921413 (+ 275us) server_negotiation.cc:407] Connection header received
0929 02:49:45.921494 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:45.921498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:45.921548 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:45.921618 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:45.922393 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.923099 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.924220 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:45.924624 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:45.924709 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:45.926644 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:45.926649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:45.926652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:45.926684 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:45.929114 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.929706 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.929709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.929711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.929833 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:45.930130 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:45.930133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:45.930135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:45.930368 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:45.930540 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:45.930754 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 02:49:45.930953 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 02:49:45.931823 10682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:46.014319 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 10559
I20260929 02:49:46.026846 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 10693
Sep 29 02:49:46 dist-test-slave-t6c1 krb5kdc[10453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:49:46.041610 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:46.027696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.4.197.130:42731)
0929 02:49:46.028262 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:46.028269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:46.028295 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:49:46.028374 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:46.028381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:46.028597 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:46.028729 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:46.029751 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:46.030824 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:46.032177 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:46.032744 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:46.032845 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:46.036588 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:46.036599 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:46.036614 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:46.036648 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:46.039303 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:46.039897 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:46.039903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:46.039906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:46.040019 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:46.040312 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:46.040315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:46.040325 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:46.040644 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:46.040866 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:46.041118 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 02:49:46.041308 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":308,"threads_started":1}
I20260929 02:49:46.042827 10816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:46.122982 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 10693
I20260929 02:49:46.135299 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 10826
Sep 29 02:49:46 dist-test-slave-t6c1 krb5kdc[10453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:49:46.149950 10976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:46.136146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.4.197.131:45359)
0929 02:49:46.136737 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:46.136744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:46.136773 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:49:46.136854 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:46.136860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:46.137066 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:46.137194 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:46.138068 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:46.139128 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:46.140271 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:46.140812 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:46.140919 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:46.144707 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:46.144718 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:46.144732 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:46.144762 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:46.147470 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:46.148104 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:46.148111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:46.148114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:46.148229 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:46.148565 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:46.148569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:46.148571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:46.148890 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:46.149171 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:46.149424 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 02:49:46.149660 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":276,"threads_started":1}
I20260929 02:49:46.151153 10949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:46.233284 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 10826
I20260929 02:49:46.245391 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 10466
I20260929 02:49:46.246516 10528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:46.346241 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 10466
2026-09-29T02:49:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3704 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:46 dist-test-slave-t6c1 krb5kdc[10981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:46 dist-test-slave-t6c1 krb5kdc[10981](info): set up 2 sockets
Sep 29 02:49:46 dist-test-slave-t6c1 krb5kdc[10981](info): commencing operation
krb5kdc: starting...
W20260929 02:49:47.335621 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:49:47 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.190.keytab.
Entry for principal oryx/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.190.keytab.
I20260929 02:49:47.533787 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:38301
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.190.keytab
--principal=oryx/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:38301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:47.845912 10994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:47.846196 10994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:47.846284 10994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:47.853329 10994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:47.853420 10994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:47.853470 10994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:47.853524 10994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:47.853566 10994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:47.862892 10994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:38301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.190.keytab
--principal=oryx/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:38301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:47.864218 10994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:47.865494 10994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:47.876782 11000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:47.876915 11002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:47.876786 10999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:47.877383 10994 server_base.cc:1034] running on GCE node
I20260929 02:49:47.878151 10994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:47.879621 10994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:47.880846 10994 hybrid_clock.cc:648] HybridClock initialized: now 1790650187880769 us; error 47 us; skew 500 ppm
I20260929 02:49:47.881186 10994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:47 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:47.890334 10994 init.cc:377] Logged in from keytab as oryx/127.4.197.190@KRBTEST.COM (short username oryx)
I20260929 02:49:47.891870 10994 webserver.cc:460] Webserver started at http://127.4.197.190:33225/ using document root <none> and password file <none>
I20260929 02:49:47.892536 10994 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:47.892637 10994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:47.892931 10994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:47.896133 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "4ae4591dcf7b454cbaed46d286281fc8"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-t6c1"
server_key: "23ec23fe57687af83db805e4d50c66e9"
server_key_iv: "f4392e93d4b457b953ecbe1410b184fd"
server_key_version: "encryptionkey@0"
I20260929 02:49:47.897018 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "4ae4591dcf7b454cbaed46d286281fc8"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-t6c1"
server_key: "23ec23fe57687af83db805e4d50c66e9"
server_key_iv: "f4392e93d4b457b953ecbe1410b184fd"
server_key_version: "encryptionkey@0"
I20260929 02:49:47.902300 10994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:49:47.905680 11009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:47.907143 10994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:47.907302 10994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "4ae4591dcf7b454cbaed46d286281fc8"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-t6c1"
server_key: "23ec23fe57687af83db805e4d50c66e9"
server_key_iv: "f4392e93d4b457b953ecbe1410b184fd"
server_key_version: "encryptionkey@0"
I20260929 02:49:47.907495 10994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:47.926527 10994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:47.930483 10994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:47.930751 10994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:47.947896 10994 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:38301
I20260929 02:49:47.947907 11060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:38301 every 8 connection(s)
I20260929 02:49:47.949433 10994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:47.951642 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 10994
I20260929 02:49:47.951851 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:47.952685 4886 external_mini_cluster.cc:1428] Setting key 09c609d47d4250d217922fceff264cc3
I20260929 02:49:47.954155 11061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:49:47 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:47.968601 11061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8: Bootstrap starting.
I20260929 02:49:47.971796 11061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:47.972811 11061 log.cc:824] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:47.975952 11061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8: No bootstrap required, opened a new log
I20260929 02:49:47.976114 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.954997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.4.197.190:38301)
0929 02:49:47.955799 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:47.955812 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:47.955858 (+ 46us) server_negotiation.cc:407] Connection header received
0929 02:49:47.957515 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:47.957540 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:47.957998 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:47.958400 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:47.960275 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.961686 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.963448 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.964134 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.964290 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.969004 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:47.969012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:47.969027 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:47.969069 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:47.971949 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:47.972635 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:47.972641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:47.972648 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:47.972829 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:47.973204 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:47.973209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:47.973213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:47.973992 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:47.974262 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:47.974685 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 02:49:47.975088 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":359,"threads_started":1}
I20260929 02:49:47.983757 11061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } }
I20260929 02:49:47.984130 11061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:47.984220 11061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae4591dcf7b454cbaed46d286281fc8, State: Initialized, Role: FOLLOWER
I20260929 02:49:47.984725 11061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } }
I20260929 02:49:47.984938 11061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:47.985052 11061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:47.985193 11061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:47.987566 11061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } }
I20260929 02:49:47.988018 11061 leader_election.cc:302] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ae4591dcf7b454cbaed46d286281fc8; no voters:
I20260929 02:49:47.988539 11061 leader_election.cc:288] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:47.988639 11066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:47.989245 11066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 4ae4591dcf7b454cbaed46d286281fc8, State: Running, Role: LEADER
I20260929 02:49:47.989790 11066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } }
I20260929 02:49:47.990532 11061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:47.992564 11067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } }
I20260929 02:49:47.992938 11067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:47.992538 11068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae4591dcf7b454cbaed46d286281fc8. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } }
I20260929 02:49:47.993104 11068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:47.994521 11074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:47.999264 11074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:48.007586 11074 catalog_manager.cc:1347] Generated new cluster ID: f1b42ab5867d47a98159071347df89ea
I20260929 02:49:48.007688 11074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:48.021473 11074 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:48.022634 11074 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:48.030215 11074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8: Generated new TSK 0
I20260929 02:49:48.031116 11074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab.
Entry for principal oryx/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab.
I20260929 02:49:48.078430 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38301
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab
--principal=oryx/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:48.393147 11087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:48.393452 11087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:48.393554 11087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:48.400739 11087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:48.400841 11087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:48.400959 11087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:48.410121 11087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab
--principal=oryx/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:48.411325 11087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:48.412531 11087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:48.425316 11092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:48.425356 11093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:48.425278 11095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:48.426017 11087 server_base.cc:1034] running on GCE node
I20260929 02:49:48.426594 11087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:48.427536 11087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:48.428731 11087 hybrid_clock.cc:648] HybridClock initialized: now 1790650188428687 us; error 34 us; skew 500 ppm
I20260929 02:49:48.429037 11087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:48 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650188, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:48.432053 11087 init.cc:377] Logged in from keytab as oryx/127.4.197.129@KRBTEST.COM (short username oryx)
I20260929 02:49:48.433483 11087 webserver.cc:460] Webserver started at http://127.4.197.129:36937/ using document root <none> and password file <none>
I20260929 02:49:48.434319 11087 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:48.434456 11087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:48.434748 11087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:48.437806 11087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "af8df5fbcb6e46e09832e71a7ef19da7"
format_stamp: "Formatted at 2026-09-29 02:49:48 on dist-test-slave-t6c1"
server_key: "30e9d9202b3395d546aabef57a7a5417"
server_key_iv: "f017e8cc735b0d753ade649b9ca74963"
server_key_version: "encryptionkey@0"
I20260929 02:49:48.438560 11087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "af8df5fbcb6e46e09832e71a7ef19da7"
format_stamp: "Formatted at 2026-09-29 02:49:48 on dist-test-slave-t6c1"
server_key: "30e9d9202b3395d546aabef57a7a5417"
server_key_iv: "f017e8cc735b0d753ade649b9ca74963"
server_key_version: "encryptionkey@0"
I20260929 02:49:48.443845 11087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:49:48.447327 11102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:48.448680 11087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:49:48.448844 11087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "af8df5fbcb6e46e09832e71a7ef19da7"
format_stamp: "Formatted at 2026-09-29 02:49:48 on dist-test-slave-t6c1"
server_key: "30e9d9202b3395d546aabef57a7a5417"
server_key_iv: "f017e8cc735b0d753ade649b9ca74963"
server_key_version: "encryptionkey@0"
I20260929 02:49:48.449038 11087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:48.460474 11087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:48.464184 11087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:48.464460 11087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:48.465418 11087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:48.466770 11087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:48.466840 11087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:48.466960 11087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:48.467012 11087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 02:49:48.496165 11087 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:33895
I20260929 02:49:48.496212 11214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:33895 every 8 connection(s)
I20260929 02:49:48.497386 11087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:48.506481 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 11087
I20260929 02:49:48.506680 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:48.507462 4886 external_mini_cluster.cc:1428] Setting key 1ac3f30a0119bfff6c8094df50507e3d
Sep 29 02:49:48 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650188, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:48.517211 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:48.499843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:55175 (local address 127.4.197.190:38301)
0929 02:49:48.500286 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:48.500292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:48.501246 (+ 954us) server_negotiation.cc:407] Connection header received
0929 02:49:48.503108 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:48.503113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:48.503177 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:48.503260 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:48.504662 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.505361 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.506993 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.507608 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.507738 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:48.511460 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:48.511487 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:48.511492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:48.511530 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:48.514406 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:48.515089 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:48.515095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:48.515098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:48.515228 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:48.515598 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:48.515602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:48.515605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:48.515901 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:48.516007 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:48.516806 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 02:49:48.516960 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":198,"threads_started":1}
I20260929 02:49:48.518540 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:48.500438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.129:55175)
0929 02:49:48.501089 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:48.501127 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:48.502819 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:48.503458 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:48.503467 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:48.503952 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:48.504508 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:48.504526 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.505475 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.505478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:48.506832 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:48.506846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.507806 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.507812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:48.508011 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:48.508771 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:48.508799 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:48.511227 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:48.514532 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:48.514539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:48.514551 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:48.514996 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:48.515326 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:48.515329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:48.515332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:48.515501 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:48.516013 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:48.516021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:48.516458 (+ 437us) client_negotiation.cc:769] Sending connection context
0929 02:49:48.516753 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 02:49:48.517243 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":245,"threads_started":1}
I20260929 02:49:48.520015 11215 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38301
I20260929 02:49:48.520273 11215 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:48.520901 11215 heartbeater.cc:511] Master 127.4.197.190:38301 requested a full tablet report, sending...
I20260929 02:49:48.523031 11026 ts_manager.cc:194] Registered new tserver with Master: af8df5fbcb6e46e09832e71a7ef19da7 (127.4.197.129:33895)
I20260929 02:49:48.525132 11026 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:55175
WARNING: no policy specified for oryx/127.4.197.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab.
Entry for principal oryx/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab.
I20260929 02:49:48.554658 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38301
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab
--principal=oryx/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:48.867450 11221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:48.867751 11221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:48.867897 11221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:48.875491 11221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:48.875605 11221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:48.875769 11221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:48.885561 11221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab
--principal=oryx/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:48.886901 11221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:48.888267 11221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:48.901175 11227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:48.901237 11226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:48.901262 11229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:48.901829 11221 server_base.cc:1034] running on GCE node
I20260929 02:49:48.902473 11221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:48.903461 11221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:48.904652 11221 hybrid_clock.cc:648] HybridClock initialized: now 1790650188904601 us; error 46 us; skew 500 ppm
I20260929 02:49:48.904950 11221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:48 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650188, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:48.907896 11221 init.cc:377] Logged in from keytab as oryx/127.4.197.130@KRBTEST.COM (short username oryx)
I20260929 02:49:48.909243 11221 webserver.cc:460] Webserver started at http://127.4.197.130:43899/ using document root <none> and password file <none>
I20260929 02:49:48.909884 11221 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:48.909996 11221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:48.910260 11221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:48.913196 11221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "770e83e4819e49fabebce41221489ed1"
format_stamp: "Formatted at 2026-09-29 02:49:48 on dist-test-slave-t6c1"
server_key: "386d137e853f8d9e6696fd342aba2b0e"
server_key_iv: "845e29f216de3d3342542ffac1e1b8f6"
server_key_version: "encryptionkey@0"
I20260929 02:49:48.913908 11221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "770e83e4819e49fabebce41221489ed1"
format_stamp: "Formatted at 2026-09-29 02:49:48 on dist-test-slave-t6c1"
server_key: "386d137e853f8d9e6696fd342aba2b0e"
server_key_iv: "845e29f216de3d3342542ffac1e1b8f6"
server_key_version: "encryptionkey@0"
I20260929 02:49:48.918733 11221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 02:49:48.921973 11236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:48.923354 11221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:48.923502 11221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "770e83e4819e49fabebce41221489ed1"
format_stamp: "Formatted at 2026-09-29 02:49:48 on dist-test-slave-t6c1"
server_key: "386d137e853f8d9e6696fd342aba2b0e"
server_key_iv: "845e29f216de3d3342542ffac1e1b8f6"
server_key_version: "encryptionkey@0"
I20260929 02:49:48.923684 11221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:48.954365 11221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:48.958643 11221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:48.958946 11221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:48.960021 11221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:48.961303 11221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:48.961380 11221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:48.961494 11221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:48.961553 11221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:48.990617 11221 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:39001
I20260929 02:49:48.990664 11348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:39001 every 8 connection(s)
I20260929 02:49:48.991918 11221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:48.994257 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 11221
I20260929 02:49:48.994437 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:48.994874 4886 external_mini_cluster.cc:1428] Setting key 12473954af15a7b44cbcd71e00900124
Sep 29 02:49:49 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650188, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:49.013899 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:48.995127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:56917 (local address 127.4.197.190:38301)
0929 02:49:48.995333 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:48.995339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:48.996209 (+ 870us) server_negotiation.cc:407] Connection header received
0929 02:49:48.998307 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:48.998312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:48.998396 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:48.998499 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:49.000371 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.001345 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.003441 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.004039 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.004158 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:49.007609 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:49.007615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:49.007620 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:49.007660 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:49.010706 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:49.011555 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:49.011561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:49.011564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:49.011727 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:49.012136 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:49.012140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:49.012143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:49.012529 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:49.012684 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:49.013336 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 02:49:49.013555 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for oryx/127.4.197.131@KRBTEST.COM; defaulting to no policy
I20260929 02:49:49.014639 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:48.995157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.130:56917)
0929 02:49:48.996045 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:48.996093 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:48.998003 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:48.998741 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:48.998751 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:48.999474 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:49.000200 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:49.000222 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.001479 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.001484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:49.003268 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:49.003286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.004251 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.004256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:49.004472 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:49.005117 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:49.005140 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:49.007353 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:49.010857 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:49.010867 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:49.010884 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:49.011424 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:49.011824 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:49.011828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:49.011832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:49.012023 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:49.012639 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:49.012645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:49.012963 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 02:49:49.013181 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 02:49:49.013565 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":297,"threads_started":1}
I20260929 02:49:49.015959 11349 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38301
I20260929 02:49:49.016233 11349 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:49.016949 11349 heartbeater.cc:511] Master 127.4.197.190:38301 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.131@KRBTEST.COM" created.
I20260929 02:49:49.018391 11026 ts_manager.cc:194] Registered new tserver with Master: 770e83e4819e49fabebce41221489ed1 (127.4.197.130:39001)
I20260929 02:49:49.019443 11026 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:56917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab.
Entry for principal oryx/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab.
I20260929 02:49:49.039449 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38301
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab
--principal=oryx/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:49.350660 11354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:49.351043 11354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:49.351184 11354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:49.358311 11354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:49.358417 11354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:49.358570 11354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:49.367944 11354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab
--principal=oryx/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:38301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:49.369163 11354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:49.370421 11354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:49.383013 11360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:49.383160 11359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:49.383124 11362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:49.383517 11354 server_base.cc:1034] running on GCE node
I20260929 02:49:49.384115 11354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:49.385057 11354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:49.386265 11354 hybrid_clock.cc:648] HybridClock initialized: now 1790650189386204 us; error 57 us; skew 500 ppm
I20260929 02:49:49.386562 11354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:49 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650189, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:49.389628 11354 init.cc:377] Logged in from keytab as oryx/127.4.197.131@KRBTEST.COM (short username oryx)
I20260929 02:49:49.391000 11354 webserver.cc:460] Webserver started at http://127.4.197.131:44139/ using document root <none> and password file <none>
I20260929 02:49:49.391651 11354 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:49.391745 11354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:49.391990 11354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:49.394685 11354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "bbc41f9a5e014994be48f46fab7d976e"
format_stamp: "Formatted at 2026-09-29 02:49:49 on dist-test-slave-t6c1"
server_key: "9c1da911584ba017b9bffad05004d6fc"
server_key_iv: "0e934a37d57f0c099f26eda5d491881d"
server_key_version: "encryptionkey@0"
I20260929 02:49:49.395430 11354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "bbc41f9a5e014994be48f46fab7d976e"
format_stamp: "Formatted at 2026-09-29 02:49:49 on dist-test-slave-t6c1"
server_key: "9c1da911584ba017b9bffad05004d6fc"
server_key_iv: "0e934a37d57f0c099f26eda5d491881d"
server_key_version: "encryptionkey@0"
I20260929 02:49:49.400146 11354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 02:49:49.403271 11369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:49.404474 11354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260929 02:49:49.404593 11354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "bbc41f9a5e014994be48f46fab7d976e"
format_stamp: "Formatted at 2026-09-29 02:49:49 on dist-test-slave-t6c1"
server_key: "9c1da911584ba017b9bffad05004d6fc"
server_key_iv: "0e934a37d57f0c099f26eda5d491881d"
server_key_version: "encryptionkey@0"
I20260929 02:49:49.404742 11354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:49.418828 11354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:49.422819 11354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:49.423085 11354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:49.423959 11354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:49.425171 11354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:49.425239 11354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:49.425325 11354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:49.425369 11354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:49.455318 11354 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:40099
I20260929 02:49:49.455363 11481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:40099 every 8 connection(s)
I20260929 02:49:49.456615 11354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:49.466701 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 11354
I20260929 02:49:49.466886 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:49.467494 4886 external_mini_cluster.cc:1428] Setting key b637833b72618a3d9395d0fa7a2efcd6
Sep 29 02:49:49 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650189, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:49.475623 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:49.458998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:53573 (local address 127.4.197.190:38301)
0929 02:49:49.459201 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:49.459207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:49.460438 (+ 1231us) server_negotiation.cc:407] Connection header received
0929 02:49:49.462510 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:49.462515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:49.462586 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:49.462674 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:49.464181 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.464886 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.466334 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.466750 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.466874 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:49.470693 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:49.470697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:49.470701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:49.470734 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:49.473077 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:49.473728 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:49.473732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:49.473734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:49.473853 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:49.474180 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:49.474183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:49.474185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:49.474442 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:49.474546 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:49.475208 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 02:49:49.475374 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:49:49.476577 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:49.459576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.131:53573)
0929 02:49:49.460277 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:49.460314 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:49.462227 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:49.462887 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:49.462895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:49.463451 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:49.464021 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:49.464038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.465008 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.465011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:49.466227 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:49.466238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.467027 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.467035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:49.467351 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:49.468033 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:49.468061 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:49.470496 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:49.473201 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:49.473207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:49.473220 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:49.473635 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:49.473928 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:49.473931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:49.473933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:49.474102 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:49.474532 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:49.474537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:49.474858 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 02:49:49.475120 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 02:49:49.475514 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":270,"threads_started":1}
I20260929 02:49:49.477844 11482 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38301
I20260929 02:49:49.478123 11482 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:49.478830 11482 heartbeater.cc:511] Master 127.4.197.190:38301 requested a full tablet report, sending...
I20260929 02:49:49.480223 11026 ts_manager.cc:194] Registered new tserver with Master: bbc41f9a5e014994be48f46fab7d976e (127.4.197.131:40099)
I20260929 02:49:49.481024 11026 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:53573
I20260929 02:49:49.482868 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.197.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.189.keytab.
Entry for principal oryx/127.4.197.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.189.keytab.
I20260929 02:49:49.527415 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.189:43749
--webserver_interface=127.4.197.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.189.keytab
--principal=oryx/127.4.197.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:38301,127.4.197.189:43749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:49:49.528206 11215 heartbeater.cc:503] Master 127.4.197.190:38301 was elected leader, sending a full tablet report...
W20260929 02:49:49.841401 11487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:49.841688 11487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:49.841869 11487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:49.849007 11487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:49.849148 11487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:49.849226 11487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:49.849303 11487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:49.849360 11487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:49.859227 11487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:38301,127.4.197.189:43749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.189.keytab
--principal=oryx/127.4.197.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.189:43749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.197.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
W20260929 02:49:49.859604 11487 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.197.190:38301,127.4.197.189:43749'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 02:49:49.860638 11487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:49.861927 11487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:49.872689 11492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:49.872717 11495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:49.872767 11493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:49.873589 11487 server_base.cc:1034] running on GCE node
I20260929 02:49:49.874332 11487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:49.875731 11487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:49.876955 11487 hybrid_clock.cc:648] HybridClock initialized: now 1790650189876885 us; error 47 us; skew 500 ppm
I20260929 02:49:49.877260 11487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:49 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650189, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:49.880412 11487 init.cc:377] Logged in from keytab as oryx/127.4.197.189@KRBTEST.COM (short username oryx)
I20260929 02:49:49.881939 11487 webserver.cc:460] Webserver started at http://127.4.197.189:44365/ using document root <none> and password file <none>
I20260929 02:49:49.882642 11487 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:49.882728 11487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:49.883067 11487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:49.885934 11487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data/instance:
uuid: "33894a7c546c4f3eaebaeaa10dc84b6e"
format_stamp: "Formatted at 2026-09-29 02:49:49 on dist-test-slave-t6c1"
server_key: "03a2844a9965adbdb3ea10bd06aac80d"
server_key_iv: "af0513db28312682a613ac787e4efe54"
server_key_version: "encryptionkey@0"
I20260929 02:49:49.886642 11487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal/instance:
uuid: "33894a7c546c4f3eaebaeaa10dc84b6e"
format_stamp: "Formatted at 2026-09-29 02:49:49 on dist-test-slave-t6c1"
server_key: "03a2844a9965adbdb3ea10bd06aac80d"
server_key_iv: "af0513db28312682a613ac787e4efe54"
server_key_version: "encryptionkey@0"
I20260929 02:49:49.891747 11487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 02:49:49.895315 11502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:49.896780 11487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:49:49.896970 11487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal
uuid: "33894a7c546c4f3eaebaeaa10dc84b6e"
format_stamp: "Formatted at 2026-09-29 02:49:49 on dist-test-slave-t6c1"
server_key: "03a2844a9965adbdb3ea10bd06aac80d"
server_key_iv: "af0513db28312682a613ac787e4efe54"
server_key_version: "encryptionkey@0"
I20260929 02:49:49.897166 11487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:49.927472 11487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:49.932018 11487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:49.932265 11487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:49.959583 11487 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.189:43749
I20260929 02:49:49.959618 11553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.189:43749 every 8 connection(s)
I20260929 02:49:49.961230 11487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
I20260929 02:49:49.965906 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 11487
I20260929 02:49:49.966054 11554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:49.977394 11554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:49:49 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.189@KRBTEST.COM
Sep 29 02:49:49 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650189, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.189@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:49.996675 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:49.968070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.4.197.189:43749)
0929 02:49:49.968808 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:49.968824 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:49.968871 (+ 47us) server_negotiation.cc:407] Connection header received
0929 02:49:49.971546 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:49.971576 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:49.972112 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:49.972591 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:49.974149 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.976205 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.978451 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.979778 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.980008 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:49.985568 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:49.985581 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:49.985602 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:49.985655 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:49.990113 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:49.992402 (+ 2289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:49.992411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:49.992421 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:49.992622 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:49.993149 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:49.993155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:49.993158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:49.993893 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:49.994169 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:49.994980 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 02:49:49.995483 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":282,"threads_started":1}
I20260929 02:49:50.002009 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:49.980721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.4.197.190:38301)
0929 02:49:49.982026 (+ 1305us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:49.982032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:49.982053 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:49:49.982119 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:49.982125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:49.982208 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:49.982325 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:49.983941 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.984803 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.986811 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.987319 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.987438 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:49.996627 (+ 9189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:49.996633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:49.996636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:49.996671 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:49.999425 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.000253 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.000259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.000262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.000424 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.000709 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.000712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.000714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.001014 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.001152 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.001587 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.001764 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":1006,"threads_started":1}
I20260929 02:49:50.002041 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:49.980653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40230)
0929 02:49:49.981356 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:49.981399 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:49.981544 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:49.982435 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:49.982443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:49.983407 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:49.983752 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:49.983767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.984944 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.984950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:49.986665 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:49.986682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:49.987507 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:49.987513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:49.987704 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:49.988341 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:49.988357 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:49.991203 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:49.999655 (+ 8452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:49.999666 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:49.999673 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.000128 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.000505 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.000508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.000510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.000632 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.001165 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.001171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.001311 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.001469 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.001746 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":300,"threads_started":1}
I20260929 02:49:50.003600 11554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650189, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.189@KRBTEST.COM for oryx/127.4.197.189@KRBTEST.COM
I20260929 02:49:50.014765 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.004285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:43749 (local address 127.0.0.1:55848)
0929 02:49:50.004452 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.004469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.004555 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.004782 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.004785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.005063 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.005207 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.005214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.006180 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.006183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.007229 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.007240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.007920 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.007926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.008065 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.008535 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.008550 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.010429 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.012941 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.012944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.012947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.013269 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.013531 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.013534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.013536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.013644 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.014090 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.014093 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.014213 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.014325 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.014510 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:49:50.014941 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.004341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.4.197.189:43749)
0929 02:49:50.004537 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.004541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.004556 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:49:50.004635 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.004639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.004690 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.004762 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.005338 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.006068 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.007345 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.007740 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.007855 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.010578 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.010582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.010585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.010611 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.012815 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.013363 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.013366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.013369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.013461 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.013720 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.013723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.013725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.013997 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.014097 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.014387 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.014623 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":104}
I20260929 02:49:50.019246 11554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: Bootstrap starting.
I20260929 02:49:50.021806 11349 heartbeater.cc:503] Master 127.4.197.190:38301 was elected leader, sending a full tablet report...
I20260929 02:49:50.022308 11554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:50.023418 11554 log.cc:824] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:50.025931 11554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: No bootstrap required, opened a new log
I20260929 02:49:50.032949 11554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } }
I20260929 02:49:50.033320 11554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:50.033416 11554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33894a7c546c4f3eaebaeaa10dc84b6e, State: Initialized, Role: FOLLOWER
I20260929 02:49:50.034030 11554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } }
I20260929 02:49:50.034861 11561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } } }
I20260929 02:49:50.035180 11561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:50.035882 11554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:49:50.043459 11572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:49:50.043565 11572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:49:50.046090 11487 master_runner.cc:428] Detected that this master 33894a7c546c4f3eaebaeaa10dc84b6e is joining an existing cluster
I20260929 02:49:50.046172 11487 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.197.189:43749
I20260929 02:49:50.065423 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.050092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.4.197.189:43749)
0929 02:49:50.050291 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.050296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.050954 (+ 658us) server_negotiation.cc:407] Connection header received
0929 02:49:50.051113 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.051118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.051199 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.051290 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.053302 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.054332 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.056204 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.056900 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.057003 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.059286 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.059292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.059297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.059337 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.062625 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.063433 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.063438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.063442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.063616 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.063982 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.063987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.063990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.064337 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.064493 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.064837 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.065045 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":24320}
I20260929 02:49:50.066411 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.049548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.4.197.190:38301)
0929 02:49:50.049721 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.049728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.050413 (+ 685us) server_negotiation.cc:407] Connection header received
0929 02:49:50.051312 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.051317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.051405 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.051469 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.052650 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.053715 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.055679 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.056267 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.056366 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.059288 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.059305 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.059309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.059346 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.062971 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.063690 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.063695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.063698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.063855 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.064713 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.064719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.064722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.065058 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.065181 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.065832 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.066065 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:49:50.066610 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.049737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:43749 (local address 127.0.0.1:55858)
0929 02:49:50.050838 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.050853 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.051003 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.051459 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.051464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.052875 (+ 1411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.053170 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.053181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.054571 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.054576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.056068 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.056080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.057245 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.057251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.057438 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.058041 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.058062 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.059104 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.062770 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.062775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.062780 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.063322 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.063731 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.063735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.063738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.063888 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.064449 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.064454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.064595 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.065776 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.066065 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"spinlock_wait_cycles":1408,"thread_start_us":808,"threads_started":1}
I20260929 02:49:50.068334 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.049685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40236)
0929 02:49:50.050308 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.050333 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.051155 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.051667 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.051671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.052088 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.052463 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.052480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.053921 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.053926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.055532 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.055546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.056634 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.056639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.056872 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.057533 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.057557 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.059104 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.063133 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.063138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.063142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.063561 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.063984 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.063988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.063992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.064587 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.065258 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.065261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.065478 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.066647 (+ 1169us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.066971 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":50,"spinlock_wait_cycles":40064,"thread_start_us":350,"threads_started":1}
WARNING: no policy specified for oryx/127.4.197.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.188@KRBTEST.COM" created.
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.190@KRBTEST.COM for oryx/127.4.197.189@KRBTEST.COM
I20260929 02:49:50.090699 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.074037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.4.197.189:43749)
0929 02:49:50.074307 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.074314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.075668 (+ 1354us) server_negotiation.cc:407] Connection header received
0929 02:49:50.075968 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.075973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.076095 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.076237 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.077698 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.078719 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.080665 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.081266 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.081402 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.085274 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.085279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.085282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.085323 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.088002 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.088675 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.088679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.088682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.088822 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.089202 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.089206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.089208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.089517 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.089649 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.090112 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.090353 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:49:50.090752 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.073924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:43749 (local address 127.0.0.1:55870)
0929 02:49:50.075513 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.075559 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.075754 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.076287 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.076295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.077207 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.077521 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.077535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.078877 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.078882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.080497 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.080513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.081460 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.081467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.081703 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.082396 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.082428 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.085075 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.088133 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.088146 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.088154 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.088551 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.088926 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.088931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.088935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.089088 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.089645 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.089653 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.089828 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.089989 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.090329 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308,"thread_start_us":1126,"threads_started":1}
I20260929 02:49:50.093639 11026 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.197.189:43749: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e"
member_type: NON_VOTER
last_known_addr {
host: "127.4.197.189"
port: 43749
}
attrs {
promote: true
}
}
dest_uuid: "4ae4591dcf7b454cbaed46d286281fc8"
cas_config_opid_index: -1
I20260929 02:49:50.094635 11026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } }
I20260929 02:49:50.098907 11585 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 33894a7c546c4f3eaebaeaa10dc84b6e (127.4.197.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } } }
I20260929 02:49:50.100158 11529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:50.103142 11585 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.197.189:43749
I20260929 02:49:50.103917 11487 master.cc:537] Master@127.4.197.189:43749 shutting down...
I20260929 02:49:50.104992 11529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 FOLLOWER]: Refusing update from remote peer 4ae4591dcf7b454cbaed46d286281fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 02:49:50.107661 11584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } } }
I20260929 02:49:50.107973 11584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:50.108415 11585 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.188.keytab.
Entry for principal oryx/127.4.197.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.188.keytab.
I20260929 02:49:50.111567 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.188:44989
--webserver_interface=127.4.197.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.188.keytab
--principal=oryx/127.4.197.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:38301,127.4.197.189:43749,127.4.197.188:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:50.111603 11010 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 02:49:50.115424 11487 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:50.115896 11487 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 02:49:50.115896 11010 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 -> Peer 33894a7c546c4f3eaebaeaa10dc84b6e (127.4.197.189:43749): Couldn't send request to peer 33894a7c546c4f3eaebaeaa10dc84b6e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:49:50.116084 11487 tablet_replica.cc:330] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: stopping tablet replica
I20260929 02:49:50.132542 11487 master.cc:552] Master@127.4.197.189:43749 shutdown complete.
I20260929 02:49:50.132660 11487 master_runner.cc:305] Clearing existing system tablet
I20260929 02:49:50.133109 11487 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:50.133211 11487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:50.136286 11487 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260929 02:49:50.138056 11590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:50.138397 11487 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:50.138665 11487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 02:49:50.138742 11487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal
uuid: "33894a7c546c4f3eaebaeaa10dc84b6e"
format_stamp: "Formatted at 2026-09-29 02:49:49 on dist-test-slave-t6c1"
server_key: "03a2844a9965adbdb3ea10bd06aac80d"
server_key_iv: "af0513db28312682a613ac787e4efe54"
server_key_version: "encryptionkey@0"
I20260929 02:49:50.143810 11487 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:50.150864 11487 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:49:50.151113 11487 log.cc:1196] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 02:49:50.151561 11487 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: Deleting consensus metadata
I20260929 02:49:50.151831 11487 master_runner.cc:315] Copying system tablet from 127.4.197.190:38301
I20260929 02:49:50.154413 11487 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: tablet copy: Beginning tablet copy session from remote peer at address 127.4.197.190:38301
I20260929 02:49:50.171394 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.156372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.4.197.190:38301)
0929 02:49:50.156679 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.156686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.157011 (+ 325us) server_negotiation.cc:407] Connection header received
0929 02:49:50.157200 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.157206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.157314 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.157421 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.158444 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.159621 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.161391 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.162046 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.162176 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.164565 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.164572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.164577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.164620 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.168090 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.169039 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.169044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.169048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.169218 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.169682 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.169688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.169691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.170063 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.170215 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.170726 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.170998 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260929 02:49:50.171481 11596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.156331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40246)
0929 02:49:50.156888 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.156911 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.157074 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.157473 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.157478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.157988 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.158285 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.158298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.159786 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.159791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.161232 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.161247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.162227 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.162234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.162462 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.163236 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.163260 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.164336 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.168283 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.168289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.168293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.168893 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.169365 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.169370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.169375 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.169568 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.170203 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.170208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.170417 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.170622 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.171058 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":352,"threads_started":1}
I20260929 02:49:50.172767 11046 tablet_copy_service.cc:140] P 4ae4591dcf7b454cbaed46d286281fc8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 33894a7c546c4f3eaebaeaa10dc84b6e ({username='oryx', principal='oryx/127.4.197.189@KRBTEST.COM'} at 127.0.0.1:40246)
I20260929 02:49:50.173102 11046 tablet_copy_service.cc:161] P 4ae4591dcf7b454cbaed46d286281fc8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 33894a7c546c4f3eaebaeaa10dc84b6e at {username='oryx', principal='oryx/127.4.197.189@KRBTEST.COM'} at 127.0.0.1:40246: session id = 33894a7c546c4f3eaebaeaa10dc84b6e-00000000000000000000000000000000
I20260929 02:49:50.177700 11046 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:49:50.180065 11487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:49:50.187067 11487 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: tablet copy: Starting download of 0 data blocks...
I20260929 02:49:50.187397 11487 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: tablet copy: Starting download of 1 WAL segments...
I20260929 02:49:50.190431 11487 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:49:50.194821 11046 tablet_copy_service.cc:342] P 4ae4591dcf7b454cbaed46d286281fc8: Request end of tablet copy session 33894a7c546c4f3eaebaeaa10dc84b6e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.197.189@KRBTEST.COM'} at 127.0.0.1:40246
I20260929 02:49:50.195081 11046 tablet_copy_service.cc:433] P 4ae4591dcf7b454cbaed46d286281fc8: ending tablet copy session 33894a7c546c4f3eaebaeaa10dc84b6e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 33894a7c546c4f3eaebaeaa10dc84b6e
I20260929 02:49:50.207082 11487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:50.209769 11597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:50.210817 11598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:50.217849 11600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:50.223891 11487 server_base.cc:1034] running on GCE node
I20260929 02:49:50.224275 11487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:50.224941 11487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:50.226166 11487 hybrid_clock.cc:648] HybridClock initialized: now 1790650190226098 us; error 36 us; skew 500 ppm
I20260929 02:49:50.226378 11487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650190, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:50.228602 11487 init.cc:377] Logged in from keytab as oryx/127.4.197.189@KRBTEST.COM (short username oryx)
I20260929 02:49:50.235939 11487 webserver.cc:460] Webserver started at http://127.4.197.189:43491/ using document root <none> and password file <none>
I20260929 02:49:50.236339 11487 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:50.236430 11487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:50.239578 11487 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:49:50.241516 11607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:50.241941 11487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260929 02:49:50.242141 11487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal
uuid: "33894a7c546c4f3eaebaeaa10dc84b6e"
format_stamp: "Formatted at 2026-09-29 02:49:49 on dist-test-slave-t6c1"
server_key: "03a2844a9965adbdb3ea10bd06aac80d"
server_key_iv: "af0513db28312682a613ac787e4efe54"
server_key_version: "encryptionkey@0"
I20260929 02:49:50.242398 11487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:50.257663 11487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:50.261734 11487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:50.284503 11487 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.189:43749
I20260929 02:49:50.284528 11658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.189:43749 every 8 connection(s)
I20260929 02:49:50.285646 11487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
I20260929 02:49:50.290853 11659 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:49:50.294659 11659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: Bootstrap starting.
I20260929 02:49:50.311357 11659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:49:50.311856 11659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: Bootstrap complete.
I20260929 02:49:50.312879 11659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } }
I20260929 02:49:50.313150 11659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 33894a7c546c4f3eaebaeaa10dc84b6e, State: Initialized, Role: LEARNER
I20260929 02:49:50.313409 11659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } }
I20260929 02:49:50.314272 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } } }
I20260929 02:49:50.314507 11659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:50.314723 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: This master's current role is: LEARNER
I20260929 02:49:50.324543 11673 catalog_manager.cc:1259] Loaded cluster ID: f1b42ab5867d47a98159071347df89ea
I20260929 02:49:50.324649 11673 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: loading cluster ID for follower catalog manager: success
I20260929 02:49:50.328540 11673 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: acquiring CA information for follower catalog manager: success
I20260929 02:49:50.331460 11673 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:49:50.483754 11482 heartbeater.cc:503] Master 127.4.197.190:38301 was elected leader, sending a full tablet report...
W20260929 02:49:50.488340 11588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:50.488616 11588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:50.488711 11588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:50.495908 11588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:50.495996 11588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:50.496064 11588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:50.496132 11588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:50.496188 11588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:50.506659 11588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:38301,127.4.197.189:43749,127.4.197.188:44989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.188.keytab
--principal=oryx/127.4.197.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.188:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.197.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:50.508003 11588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:50.509644 11588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:50.523243 11681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:50.523188 11678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:50.523212 11679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:50.524139 11588 server_base.cc:1034] running on GCE node
I20260929 02:49:50.524999 11588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:50.526602 11588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:50.528208 11588 hybrid_clock.cc:648] HybridClock initialized: now 1790650190527746 us; error 62 us; skew 500 ppm
I20260929 02:49:50.528561 11588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650190, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:50.531984 11588 init.cc:377] Logged in from keytab as oryx/127.4.197.188@KRBTEST.COM (short username oryx)
I20260929 02:49:50.533437 11588 webserver.cc:460] Webserver started at http://127.4.197.188:38133/ using document root <none> and password file <none>
I20260929 02:49:50.534083 11588 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:50.534183 11588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:50.534439 11588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:50.537411 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data/instance:
uuid: "4618292714304d2190f0fb80c1582564"
format_stamp: "Formatted at 2026-09-29 02:49:50 on dist-test-slave-t6c1"
server_key: "38a2771f457027fc335412843c0de2e7"
server_key_iv: "d9520e3d1e553994d3ab6a86c2d7c091"
server_key_version: "encryptionkey@0"
I20260929 02:49:50.538115 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal/instance:
uuid: "4618292714304d2190f0fb80c1582564"
format_stamp: "Formatted at 2026-09-29 02:49:50 on dist-test-slave-t6c1"
server_key: "38a2771f457027fc335412843c0de2e7"
server_key_iv: "d9520e3d1e553994d3ab6a86c2d7c091"
server_key_version: "encryptionkey@0"
I20260929 02:49:50.543203 11588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 02:49:50.546425 11688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:50.547830 11588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:50.547995 11588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal
uuid: "4618292714304d2190f0fb80c1582564"
format_stamp: "Formatted at 2026-09-29 02:49:50 on dist-test-slave-t6c1"
server_key: "38a2771f457027fc335412843c0de2e7"
server_key_iv: "d9520e3d1e553994d3ab6a86c2d7c091"
server_key_version: "encryptionkey@0"
I20260929 02:49:50.548177 11588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:50.567180 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.560574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.4.197.189:43749)
0929 02:49:50.561139 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.561145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.561168 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:49:50.561243 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.561247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.561341 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.561523 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:50.561950 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.562969 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.565319 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.566149 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.566263 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.566438 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.566759 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.566875 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":286,"threads_started":1}
I20260929 02:49:50.567183 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.560092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:43749 (local address 127.0.0.1:55874)
0929 02:49:50.560264 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.560287 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.560414 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.561575 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.561581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.561615 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:50.561807 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.561817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.563130 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.563136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.565137 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.565155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.566304 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.566310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.566550 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.566633 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.566719 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.566875 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:49:50.568349 11634 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 02:49:50.568908 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae4591dcf7b454cbaed46d286281fc8. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } } }
I20260929 02:49:50.569183 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: This master's current role is: LEARNER
I20260929 02:49:50.608014 11588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:50.611959 11588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:50.612670 11588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:50.629709 11588 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.188:44989
I20260929 02:49:50.629727 11740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.188:44989 every 8 connection(s)
I20260929 02:49:50.631166 11588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
I20260929 02:49:50.634065 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 11588
I20260929 02:49:50.635638 11741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:50.643762 11741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.188@KRBTEST.COM
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650190, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.188@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:50.656420 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.636660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.4.197.188:44989)
0929 02:49:50.637461 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.637473 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.637514 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:49:50.639029 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.639054 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.639497 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.639877 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.641336 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.642682 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.644144 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.644861 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.645016 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.649303 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.649310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.649327 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.649357 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.652459 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.653131 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.653139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.653147 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.653340 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.653766 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.653770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.653774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.654332 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.654528 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.655066 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.655428 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":351,"threads_started":1}
I20260929 02:49:50.660588 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.646644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40262)
0929 02:49:50.647347 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.647393 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.647532 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.647904 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.647912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.648574 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.648789 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.648798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.649810 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.649814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.651476 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.651489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.652205 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.652212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.652409 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.652996 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.653012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.655250 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.658099 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.658108 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.658114 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.658604 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.658968 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.658971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.658973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.659104 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.659563 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.659568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.659692 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.659835 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.660028 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":349,"threads_started":1}
I20260929 02:49:50.660732 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.646321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.4.197.190:38301)
0929 02:49:50.646529 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.646535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.647486 (+ 951us) server_negotiation.cc:407] Connection header received
0929 02:49:50.647650 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.647656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.647739 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.647827 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.648933 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.649680 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.651619 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.652038 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.652128 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.655402 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.655406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.655410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.655441 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.657985 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.658716 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.658720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.658722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.658836 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.659210 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.659215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.659218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.659470 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.659577 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.659913 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.660060 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:49:50.661808 11741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650190, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.188@KRBTEST.COM for oryx/127.4.197.189@KRBTEST.COM
I20260929 02:49:50.672416 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.662488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:43749 (local address 127.0.0.1:55886)
0929 02:49:50.662636 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.662650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.662727 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.663080 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.663084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.663362 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.663470 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.663477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.664363 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.664368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.665219 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.665229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.665853 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.665858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.666005 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.666443 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.666457 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.668142 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.670636 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.670639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.670641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.671041 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.671323 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.671326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.671328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.671443 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.671821 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.671824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.671927 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.672023 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.672215 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 02:49:50.672484 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.662639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.4.197.189:43749)
0929 02:49:50.662829 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.662833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.662847 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:49:50.662890 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.662894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.662974 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.663052 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.663589 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.664243 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.665347 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.665722 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.665822 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.668287 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.668291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.668295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.668329 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.670523 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.671123 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.671126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.671128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.671240 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.671514 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.671517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.671519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.671742 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.671859 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.672103 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.672262 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:49:50.673326 11741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650190, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.188@KRBTEST.COM for oryx/127.4.197.188@KRBTEST.COM
I20260929 02:49:50.684414 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.673837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:44989 (local address 127.0.0.1:37376)
0929 02:49:50.673976 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.673993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.674074 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.674283 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.674286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.674564 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.674708 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.674715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.675602 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.675605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.676430 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.676441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.677101 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.677106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.677314 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.677817 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.677834 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.679599 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.682391 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.682394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.682397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.682734 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.683092 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.683095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.683098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.683243 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.683656 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.683659 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.683799 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.683903 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.684091 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 02:49:50.684482 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.673963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.4.197.188:44989)
0929 02:49:50.674085 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.674089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.674121 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:49:50.674153 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.674156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.674202 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.674261 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.674807 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.675493 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.676571 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.676963 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.677058 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.679739 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.679743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.679747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.679772 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.682264 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.682837 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.682841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.682843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.683000 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.683322 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.683325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.683327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.683571 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.683671 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.683974 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.684218 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":44}
I20260929 02:49:50.688848 11741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Bootstrap starting.
I20260929 02:49:50.691884 11741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:50.692832 11741 log.cc:824] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:50.695173 11741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: No bootstrap required, opened a new log
I20260929 02:49:50.702128 11741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } }
I20260929 02:49:50.702431 11741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:50.702525 11741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4618292714304d2190f0fb80c1582564, State: Initialized, Role: FOLLOWER
I20260929 02:49:50.703034 11741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } }
I20260929 02:49:50.703785 11747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } } }
I20260929 02:49:50.704061 11747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:50.704622 11741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:50.710745 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 11087
W20260929 02:49:50.713079 11758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:49:50.713191 11758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:49:50.717373 11588 master_runner.cc:428] Detected that this master 4618292714304d2190f0fb80c1582564 is joining an existing cluster
I20260929 02:49:50.717478 11588 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.197.188:44989
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.129@KRBTEST.COM
I20260929 02:49:50.735903 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.712357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.4.197.129:33895)
0929 02:49:50.713707 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.713715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.713744 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:49:50.713816 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.713823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.714061 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.714237 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.715808 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.717911 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.719835 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.720777 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.720905 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.726959 (+ 6054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.726982 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.727004 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.727063 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.731250 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.732319 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.732328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.732332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.732516 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.733006 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.733012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.733015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.733479 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.733899 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.735152 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.735447 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":1058,"threads_started":1}
I20260929 02:49:50.737713 11209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:50.744304 11747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:50.744607 11747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } }
I20260929 02:49:50.746299 11747 leader_election.cc:288] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ae4591dcf7b454cbaed46d286281fc8 (127.4.197.190:38301), 33894a7c546c4f3eaebaeaa10dc84b6e (127.4.197.189:43749)
I20260929 02:49:50.746867 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4618292714304d2190f0fb80c1582564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae4591dcf7b454cbaed46d286281fc8" is_pre_election: true
I20260929 02:49:50.747035 11634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4618292714304d2190f0fb80c1582564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" is_pre_election: true
I20260929 02:49:50.748093 11689 leader_election.cc:302] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4618292714304d2190f0fb80c1582564; no voters: 33894a7c546c4f3eaebaeaa10dc84b6e, 4ae4591dcf7b454cbaed46d286281fc8
I20260929 02:49:50.748790 11747 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:49:50.753684 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.722725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.4.197.189:43749)
0929 02:49:50.722908 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.722949 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.736949 (+ 14000us) server_negotiation.cc:407] Connection header received
0929 02:49:50.737188 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.737194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.737275 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.737342 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.739176 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.739866 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.742006 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.742659 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.742751 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.745282 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.745289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.745294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.745335 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.750118 (+ 4783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.750912 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.750936 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.750940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.751098 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.751516 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.751521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.751524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.751877 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.752004 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.752834 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.753070 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:49:50.754704 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.722285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:43749 (local address 127.0.0.1:55902)
0929 02:49:50.736838 (+ 14553us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.736857 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.737052 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.737546 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.737550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.738130 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.739008 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.739039 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.740004 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.740009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.741847 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.741861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.743056 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.743062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.743234 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.743935 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.743961 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.745077 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.750274 (+ 5197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.750279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.750282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.750768 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.751236 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.751241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.751244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.751405 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.752080 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.752085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.752557 (+ 472us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.753941 (+ 1384us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.754260 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14414,"thread_start_us":14975,"threads_started":1}
I20260929 02:49:50.758855 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.722747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.4.197.190:38301)
0929 02:49:50.722971 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.722977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.731533 (+ 8556us) server_negotiation.cc:407] Connection header received
0929 02:49:50.731811 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.731817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.731902 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.731974 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.733536 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.734749 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.744315 (+ 9566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.744945 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.745046 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.751433 (+ 6387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.751441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.751447 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.751495 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.754988 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.755799 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.755805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.755809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.755978 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.757057 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.757062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.757066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.757452 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.757588 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.758207 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.758482 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:49:50.759991 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.722150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40270)
0929 02:49:50.731416 (+ 9266us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.731436 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.731651 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.732219 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.732224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.732957 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.733343 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.733355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.734995 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.735001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.736794 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.736813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.745434 (+ 8621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.745440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.745608 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.746368 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.746393 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.747365 (+ 972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.755159 (+ 7794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.755165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.755170 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.755654 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.756123 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.756128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.756132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.756912 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.757670 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.757675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.757912 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.759134 (+ 1222us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.759514 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9102,"thread_start_us":8909,"threads_started":1}
I20260929 02:49:50.772190 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.731681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.4.197.188:44989)
0929 02:49:50.731874 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.731880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.756407 (+ 24527us) server_negotiation.cc:407] Connection header received
0929 02:49:50.756566 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.756571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.756624 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.756674 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.757627 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.758727 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.760475 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.761186 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.762264 (+ 1078us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.764601 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.764608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.764612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.764642 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.768781 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.769562 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.769567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.769571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.769736 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.770546 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.770550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.770553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.770945 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.771092 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.771602 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.771827 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:49:50.773563 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.731993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:44989 (local address 127.0.0.1:37386)
0929 02:49:50.756335 (+ 24342us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.756354 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.756474 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.756849 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.756854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.757331 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.757497 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.757507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.758952 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.758957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.760348 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.760361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.762518 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.762523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.762702 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.763364 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.763385 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.764423 (+ 1038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.768933 (+ 4510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.768938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.768943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.769444 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.769874 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.769878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.769882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.770442 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.771157 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.771164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.771308 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.772894 (+ 1586us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.773188 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24240,"thread_start_us":31117,"threads_started":1}
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.190@KRBTEST.COM for oryx/127.4.197.188@KRBTEST.COM
I20260929 02:49:50.789577 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.769153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:44989 (local address 127.0.0.1:37398)
0929 02:49:50.769479 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.769497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.769592 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.772743 (+ 3151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.772748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.773208 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.773394 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.773407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.774824 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.774830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.776270 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.776288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.777319 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.777325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.777493 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.778699 (+ 1206us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.778723 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.781829 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.785964 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.785970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.785975 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.786460 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.786971 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.786976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.786980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.787129 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.787755 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.787761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.787940 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.788062 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.788357 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260929 02:49:50.790686 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.769782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.4.197.188:44989)
0929 02:49:50.772412 (+ 2630us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.772417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.772435 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:49:50.772521 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.772526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.772603 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.772707 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.773559 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.774673 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.776444 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.777143 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.778370 (+ 1227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.782044 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.782051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.782056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.782094 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.785784 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.786628 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.786634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.786638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.786788 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.787275 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.787280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.787284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.787627 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.789968 (+ 2341us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.790086 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.790330 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2534,"thread_start_us":18448,"threads_started":1}
I20260929 02:49:50.792088 11026 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.197.188:44989: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4618292714304d2190f0fb80c1582564"
member_type: NON_VOTER
last_known_addr {
host: "127.4.197.188"
port: 44989
}
attrs {
promote: true
}
}
dest_uuid: "4ae4591dcf7b454cbaed46d286281fc8"
cas_config_opid_index: 5
I20260929 02:49:50.793136 11026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } }
I20260929 02:49:50.795526 11634 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 LEARNER]: Refusing update from remote peer 4ae4591dcf7b454cbaed46d286281fc8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:49:50.795601 11716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:50.794909 11585 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4618292714304d2190f0fb80c1582564 (127.4.197.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:50.803511 11716 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 FOLLOWER]: Refusing update from remote peer 4ae4591dcf7b454cbaed46d286281fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:49:50.810326 11770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 02:49:50.824847 11771 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:50.825562 11585 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.197.188:44989
I20260929 02:49:50.825560 11634 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4618292714304d2190f0fb80c1582564 (127.4.197.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:50.826252 11588 master.cc:537] Master@127.4.197.188:44989 shutting down...
I20260929 02:49:50.831713 11770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:50.832053 11770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:50.838583 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: NON_VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: true } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:50.838907 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: This master's current role is: LEARNER
I20260929 02:49:50.839377 11585 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8: attempting to promote NON_VOTER 33894a7c546c4f3eaebaeaa10dc84b6e to VOTER
I20260929 02:49:50.840201 11585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } }
I20260929 02:49:50.843048 11634 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 LEARNER]: Refusing update from remote peer 4ae4591dcf7b454cbaed46d286281fc8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 02:49:50.843634 11585 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 02:49:50.846292 11772 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 33894a7c546c4f3eaebaeaa10dc84b6e (127.4.197.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:50.851990 11633 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 33894a7c546c4f3eaebaeaa10dc84b6e (127.4.197.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:50.856048 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:50.856328 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:50.859751 11772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:50.860108 11772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:50.860427 11771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:50.860697 11771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: This master's current role is: LEADER
W20260929 02:49:50.863001 11011 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 -> Peer 4618292714304d2190f0fb80c1582564 (127.4.197.188:44989): Couldn't send request to peer 4618292714304d2190f0fb80c1582564. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:49:50.865032 11588 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:50.865455 11588 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:50.865653 11588 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: stopping tablet replica
I20260929 02:49:50.872779 11588 master.cc:552] Master@127.4.197.188:44989 shutdown complete.
I20260929 02:49:50.872877 11588 master_runner.cc:305] Clearing existing system tablet
I20260929 02:49:50.873281 11588 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:50.873383 11588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:50.877357 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 11087
I20260929 02:49:50.877431 11588 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:50.879210 11778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:50.879630 11588 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:50.879953 11588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 02:49:50.880080 11588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal
uuid: "4618292714304d2190f0fb80c1582564"
format_stamp: "Formatted at 2026-09-29 02:49:50 on dist-test-slave-t6c1"
server_key: "38a2771f457027fc335412843c0de2e7"
server_key_iv: "d9520e3d1e553994d3ab6a86c2d7c091"
server_key_version: "encryptionkey@0"
I20260929 02:49:50.885123 11588 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:50.892035 11588 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:49:50.892244 11588 log.cc:1196] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 02:49:50.892731 11588 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Deleting consensus metadata
I20260929 02:49:50.893011 11588 master_runner.cc:315] Copying system tablet from 127.4.197.190:38301
I20260929 02:49:50.896339 11588 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: tablet copy: Beginning tablet copy session from remote peer at address 127.4.197.190:38301
I20260929 02:49:50.898427 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:33895
--local_ip_for_outbound_sockets=127.4.197.129
--tserver_master_addrs=127.4.197.190:38301
--webserver_port=36937
--webserver_interface=127.4.197.129
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab
--principal=oryx/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.197.190:38301,127.4.197.189:43749
--tserver_master_addrs=127.4.197.190:38301,127.4.197.189:43749,127.4.197.188:44989 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:49:50.921730 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.897180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.4.197.190:38301)
0929 02:49:50.897506 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:50.897512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:50.907512 (+ 10000us) server_negotiation.cc:407] Connection header received
0929 02:49:50.907809 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:50.907815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:50.907894 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:50.907960 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:50.909114 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.910218 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.912212 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.912851 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.912957 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.915454 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:50.915461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:50.915466 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:50.915506 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:50.918821 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.919594 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.919600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.919604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.919749 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:50.920191 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:50.920196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:50.920200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:50.920526 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:50.920647 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:50.921163 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 02:49:50.921382 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260929 02:49:50.921814 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:50.897114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40286)
0929 02:49:50.907362 (+ 10248us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:50.907386 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:50.907646 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:50.908216 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:50.908221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:50.908689 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:50.908947 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.908959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.910409 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.910413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.912005 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:50.912075 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:50.913250 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:50.913261 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:50.913460 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:50.914136 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:50.914159 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:50.915239 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:50.918962 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.918967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.918972 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.919456 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.919885 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:50.919889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:50.919893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:50.920068 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:50.920718 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:50.920722 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:50.920890 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 02:49:50.921024 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:49:50.921356 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10087,"thread_start_us":9958,"threads_started":1}
I20260929 02:49:50.922405 11046 tablet_copy_service.cc:140] P 4ae4591dcf7b454cbaed46d286281fc8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4618292714304d2190f0fb80c1582564 ({username='oryx', principal='oryx/127.4.197.188@KRBTEST.COM'} at 127.0.0.1:40286)
I20260929 02:49:50.922582 11046 tablet_copy_service.cc:161] P 4ae4591dcf7b454cbaed46d286281fc8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4618292714304d2190f0fb80c1582564 at {username='oryx', principal='oryx/127.4.197.188@KRBTEST.COM'} at 127.0.0.1:40286: session id = 4618292714304d2190f0fb80c1582564-00000000000000000000000000000000
I20260929 02:49:50.926251 11046 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:49:50.928261 11588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:49:50.935017 11588 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: tablet copy: Starting download of 0 data blocks...
I20260929 02:49:50.935310 11588 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: tablet copy: Starting download of 1 WAL segments...
I20260929 02:49:50.937726 11588 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:49:50.942001 11046 tablet_copy_service.cc:342] P 4ae4591dcf7b454cbaed46d286281fc8: Request end of tablet copy session 4618292714304d2190f0fb80c1582564-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.197.188@KRBTEST.COM'} at 127.0.0.1:40286
I20260929 02:49:50.942150 11046 tablet_copy_service.cc:433] P 4ae4591dcf7b454cbaed46d286281fc8: ending tablet copy session 4618292714304d2190f0fb80c1582564-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4618292714304d2190f0fb80c1582564
I20260929 02:49:50.955917 11588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:50.972455 11786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:50.976220 11787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:50.992385 11789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:50.995711 11588 server_base.cc:1034] running on GCE node
I20260929 02:49:50.996090 11588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:50.996750 11588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:50.997978 11588 hybrid_clock.cc:648] HybridClock initialized: now 1790650190997935 us; error 42 us; skew 500 ppm
I20260929 02:49:50.998289 11588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:50 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650190, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:51.000568 11588 init.cc:377] Logged in from keytab as oryx/127.4.197.188@KRBTEST.COM (short username oryx)
I20260929 02:49:51.003829 11588 webserver.cc:460] Webserver started at http://127.4.197.188:35509/ using document root <none> and password file <none>
I20260929 02:49:51.004168 11588 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:51.004256 11588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:51.007253 11588 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:49:51.009058 11796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:51.009444 11588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 02:49:51.009590 11588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal
uuid: "4618292714304d2190f0fb80c1582564"
format_stamp: "Formatted at 2026-09-29 02:49:50 on dist-test-slave-t6c1"
server_key: "38a2771f457027fc335412843c0de2e7"
server_key_iv: "d9520e3d1e553994d3ab6a86c2d7c091"
server_key_version: "encryptionkey@0"
I20260929 02:49:51.009807 11588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:51.049168 11588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:51.053484 11588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:51.079890 11588 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.188:44989
I20260929 02:49:51.079960 11847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.188:44989 every 8 connection(s)
I20260929 02:49:51.081068 11588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
I20260929 02:49:51.086397 11848 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:49:51.090294 11848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Bootstrap starting.
I20260929 02:49:51.109272 11848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:49:51.109809 11848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Bootstrap complete.
I20260929 02:49:51.110970 11848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } }
I20260929 02:49:51.111266 11848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4618292714304d2190f0fb80c1582564, State: Initialized, Role: LEARNER
I20260929 02:49:51.111526 11848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } }
I20260929 02:49:51.112617 11848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:51.112344 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:51.112948 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:49:51.123566 11862 catalog_manager.cc:1259] Loaded cluster ID: f1b42ab5867d47a98159071347df89ea
I20260929 02:49:51.123660 11862 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: loading cluster ID for follower catalog manager: success
I20260929 02:49:51.127185 11862 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: acquiring CA information for follower catalog manager: success
I20260929 02:49:51.129825 11862 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:49:51.270422 11785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:51.270702 11785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:51.270805 11785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:51.278535 11785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:51.278647 11785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:51.278765 11785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:51.291544 11785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab
--principal=oryx/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:33895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=36937
--tserver_master_addrs=127.4.197.190:38301,127.4.197.189:43749,127.4.197.188:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:51.293061 11785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:51.294643 11785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:51.308708 11868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:51.308758 11870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:51.308736 11867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:51.309821 11785 server_base.cc:1034] running on GCE node
I20260929 02:49:51.310519 11785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:51.311553 11785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:51.312743 11785 hybrid_clock.cc:648] HybridClock initialized: now 1790650191312695 us; error 38 us; skew 500 ppm
I20260929 02:49:51.313062 11785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:51 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650191, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:51.316268 11785 init.cc:377] Logged in from keytab as oryx/127.4.197.129@KRBTEST.COM (short username oryx)
I20260929 02:49:51.317729 11785 webserver.cc:460] Webserver started at http://127.4.197.129:36937/ using document root <none> and password file <none>
I20260929 02:49:51.318352 11785 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:51.318451 11785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:51.324244 11785 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:49:51.327098 11877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:51.328511 11785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:51.328655 11785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "af8df5fbcb6e46e09832e71a7ef19da7"
format_stamp: "Formatted at 2026-09-29 02:49:48 on dist-test-slave-t6c1"
server_key: "30e9d9202b3395d546aabef57a7a5417"
server_key_iv: "f017e8cc735b0d753ade649b9ca74963"
server_key_version: "encryptionkey@0"
I20260929 02:49:51.329576 11785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:51.341441 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:51.335421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:44989 (local address 127.0.0.1:37404)
0929 02:49:51.335845 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:51.335864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:51.335991 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:51.336373 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:51.336378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:51.336400 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:51.336593 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:51.336602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.337753 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.337758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:51.339658 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:51.339674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.340711 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.340716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:51.340888 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:51.340920 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:49:51.340997 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:49:51.341124 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":192,"threads_started":1}
I20260929 02:49:51.341805 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:51.335548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.4.197.188:44989)
0929 02:49:51.336096 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:51.336101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:51.336118 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:49:51.336177 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:51.336181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:51.336244 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:51.336339 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:51.336705 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.337641 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.339808 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.340559 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.340662 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:51.340797 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:51.341031 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 02:49:51.341539 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":336,"threads_started":1}
I20260929 02:49:51.342578 11823 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 02:49:51.343144 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae4591dcf7b454cbaed46d286281fc8. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:51.343400 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:49:51.354984 11785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:51.359472 11785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:51.359719 11785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:51.360658 11785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:51.362097 11785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:51.362181 11785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:51.362265 11785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:51.362334 11785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:51.394133 11785 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:33895
I20260929 02:49:51.394191 11991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:33895 every 8 connection(s)
I20260929 02:49:51.395558 11785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:51.399601 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 11785
I20260929 02:49:51.399941 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 11221
Sep 29 02:49:51 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.130@KRBTEST.COM
Sep 29 02:49:51 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650191, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.189@KRBTEST.COM
Sep 29 02:49:51 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650191, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.188@KRBTEST.COM
Sep 29 02:49:51 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650191, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:51.449110 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:51.428833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:39185 (local address 127.4.197.188:44989)
0929 02:49:51.429096 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:51.429102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:51.430369 (+ 1267us) server_negotiation.cc:407] Connection header received
0929 02:49:51.430626 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:51.430631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:51.430707 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:51.430775 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:51.432077 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.433169 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.435506 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.436115 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.436210 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:51.441944 (+ 5734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:51.441950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:51.441955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:51.441998 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:51.445124 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:51.446270 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:51.446275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:51.446279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:51.446429 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:51.447001 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:51.447006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:51.447009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:51.447365 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:51.447491 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:51.448532 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0929 02:49:51.448773 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260929 02:49:51.449110 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:51.399245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:58601 (local address 127.4.197.190:38301)
0929 02:49:51.399463 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:51.399470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:51.427838 (+ 28368us) server_negotiation.cc:407] Connection header received
0929 02:49:51.428065 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:51.428072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:51.428162 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:51.428261 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:51.431475 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.432601 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.435527 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.436153 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.436261 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:51.442380 (+ 6119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:51.442388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:51.442392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:51.442438 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:51.446133 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:51.446904 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:51.446910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:51.446939 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:51.447089 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:51.447497 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:51.447502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:51.447505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:51.447817 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:51.447937 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:51.448532 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 02:49:51.448773 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 02:49:51.449167 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:51.398788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:51357 (local address 127.4.197.189:43749)
0929 02:49:51.428292 (+ 29504us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:51.428298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:51.428334 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:49:51.428393 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:51.428398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:51.428488 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:51.428591 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:51.431468 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.432961 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.435509 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.436155 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.436237 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:51.441651 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:51.441659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:51.441664 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:51.441708 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:51.445147 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:51.446295 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:51.446302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:51.446306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:51.446499 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:51.447023 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:51.447028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:51.447031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:51.447381 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:51.447516 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:51.448543 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0929 02:49:51.448767 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29375,"thread_start_us":29269,"threads_started":1}
I20260929 02:49:51.451429 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:51.429261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:44989 (local address 127.4.197.129:39185)
0929 02:49:51.430264 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:51.430284 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:51.430483 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:51.431009 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:51.431014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:51.431636 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:51.431902 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:51.431913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.433342 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.433346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:51.435361 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:51.435370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.436576 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.436584 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:51.437025 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:51.438553 (+ 1528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:51.438575 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:51.441682 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:51.445303 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:51.445314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:51.445334 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:51.446125 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:51.446570 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:51.446575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:51.446578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:51.446853 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:51.447573 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:51.447582 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:51.448103 (+ 521us) client_negotiation.cc:769] Sending connection context
0929 02:49:51.449342 (+ 1239us) client_negotiation.cc:240] Negotiation successful
0929 02:49:51.449918 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":389,"spinlock_wait_cycles":61184,"thread_start_us":290,"threads_started":1}
I20260929 02:49:51.451476 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:51.399592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:43749 (local address 127.4.197.129:51357)
0929 02:49:51.422316 (+ 22724us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:51.422403 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:51.426839 (+ 4436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:51.428783 (+ 1944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:51.428788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:51.430352 (+ 1564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:51.431245 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:51.431271 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.433239 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.433244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:51.435338 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:51.435354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.436635 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.436642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:51.437068 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:51.438571 (+ 1503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:51.438597 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:51.441337 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:51.445322 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:51.445327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:51.445338 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:51.446110 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:51.446681 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:51.446685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:51.446689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:51.446867 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:51.447596 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:51.447601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:51.448114 (+ 513us) client_negotiation.cc:769] Sending connection context
0929 02:49:51.449398 (+ 1284us) client_negotiation.cc:240] Negotiation successful
0929 02:49:51.450061 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22332,"mutex_wait_us":569,"spinlock_wait_cycles":67200,"thread_start_us":23558,"threads_started":1}
I20260929 02:49:51.452222 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:51.399592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.129:58601)
0929 02:49:51.427714 (+ 28122us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:51.427736 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:51.427875 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:51.428592 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:51.428606 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:51.429748 (+ 1142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:51.431238 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:51.431271 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.432831 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.432837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:51.435350 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:51.435365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.436580 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.436586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:51.437040 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:51.437818 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:51.437851 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:51.442142 (+ 4291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:51.446269 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:51.446274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:51.446277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:51.446760 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:51.447227 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:51.447231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:51.447235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:51.447381 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:51.448005 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:51.448012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:51.448130 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 02:49:51.449871 (+ 1741us) client_negotiation.cc:240] Negotiation successful
0929 02:49:51.450170 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27952,"mutex_wait_us":353,"spinlock_wait_cycles":44928,"thread_start_us":27628,"threads_started":1}
I20260929 02:49:51.452788 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:51.400944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.4.197.130:39001)
0929 02:49:51.420952 (+ 20008us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:51.420963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:51.424902 (+ 3939us) server_negotiation.cc:407] Connection header received
0929 02:49:51.425205 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:51.425215 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:51.425524 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:51.425658 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:51.427988 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.429686 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.433403 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:51.434381 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:51.434513 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:51.444899 (+ 10386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:51.444921 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:51.444940 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:51.444988 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:51.449160 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:51.450252 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:51.450262 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:51.450266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:51.450427 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:51.450856 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:51.450861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:51.450864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:51.451325 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:51.451601 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:51.452157 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 02:49:51.452421 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19771,"thread_start_us":20034,"threads_started":1}
I20260929 02:49:51.453022 11994 heartbeater.cc:347] Connected to a master server at 127.4.197.189:43749
I20260929 02:49:51.453127 11992 heartbeater.cc:347] Connected to a master server at 127.4.197.188:44989
I20260929 02:49:51.453348 11992 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:51.453666 11993 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38301
I20260929 02:49:51.453801 11993 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:51.454202 11992 heartbeater.cc:511] Master 127.4.197.188:44989 requested a full tablet report, sending...
I20260929 02:49:51.454257 11993 heartbeater.cc:511] Master 127.4.197.190:38301 requested a full tablet report, sending...
I20260929 02:49:51.453352 11994 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:51.455240 11994 heartbeater.cc:511] Master 127.4.197.189:43749 requested a full tablet report, sending...
I20260929 02:49:51.455271 11344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:51.457168 11624 ts_manager.cc:194] Registered new tserver with Master: af8df5fbcb6e46e09832e71a7ef19da7 (127.4.197.129:33895)
I20260929 02:49:51.457486 11813 ts_manager.cc:194] Registered new tserver with Master: af8df5fbcb6e46e09832e71a7ef19da7 (127.4.197.129:33895)
I20260929 02:49:51.459357 11026 ts_manager.cc:194] Re-registered known tserver with Master: af8df5fbcb6e46e09832e71a7ef19da7 (127.4.197.129:33895)
I20260929 02:49:51.460443 11026 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:58601
I20260929 02:49:51.543403 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 11221
I20260929 02:49:51.555768 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:39001
--local_ip_for_outbound_sockets=127.4.197.130
--tserver_master_addrs=127.4.197.190:38301
--webserver_port=43899
--webserver_interface=127.4.197.130
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab
--principal=oryx/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.197.190:38301,127.4.197.189:43749
--tserver_master_addrs=127.4.197.190:38301,127.4.197.189:43749,127.4.197.188:44989 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:51.870956 12003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:51.871248 12003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:51.871368 12003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:51.878638 12003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:51.878756 12003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:51.878931 12003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:51.888633 12003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab
--principal=oryx/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:39001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=43899
--tserver_master_addrs=127.4.197.190:38301,127.4.197.189:43749,127.4.197.188:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:51.889909 12003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:51.891323 12003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:51.904109 12009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:51.904130 12011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:51.904224 12008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:51.904783 12003 server_base.cc:1034] running on GCE node
I20260929 02:49:51.905396 12003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:51.906316 12003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:51.907289 11776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: NON_VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: true } } }
I20260929 02:49:51.907583 12003 hybrid_clock.cc:648] HybridClock initialized: now 1790650191907452 us; error 56 us; skew 500 ppm
I20260929 02:49:51.907622 11776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:51.907977 12003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:49:51.908763 12014 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8: attempting to promote NON_VOTER 4618292714304d2190f0fb80c1582564 to VOTER
I20260929 02:49:51.909480 12014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } }
Sep 29 02:49:51 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650191, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:51.911526 12003 init.cc:377] Logged in from keytab as oryx/127.4.197.130@KRBTEST.COM (short username oryx)
I20260929 02:49:51.922571 12003 webserver.cc:460] Webserver started at http://127.4.197.130:43899/ using document root <none> and password file <none>
I20260929 02:49:51.923238 11823 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 LEARNER]: Refusing update from remote peer 4ae4591dcf7b454cbaed46d286281fc8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 02:49:51.923470 12003 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:51.923638 12003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:51.923619 11634 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 FOLLOWER]: Refusing update from remote peer 4ae4591dcf7b454cbaed46d286281fc8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 02:49:51.923918 12014 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 02:49:51.924463 12015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 02:49:51.933022 12014 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4618292714304d2190f0fb80c1582564 (127.4.197.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:51.934096 11823 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4618292714304d2190f0fb80c1582564 (127.4.197.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:51.934774 11634 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4618292714304d2190f0fb80c1582564 (127.4.197.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:51.935338 12003 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.006s sys 0.005s
I20260929 02:49:51.939209 12024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:51.939491 12024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:51.939647 12026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:51.940014 12026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:51.940356 12025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:51.941141 12015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:51.941365 12015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:51.941609 12014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:51.941798 12014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:51.941694 12015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ae4591dcf7b454cbaed46d286281fc8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:51.941939 12015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:51.942224 12003 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:49:51.942356 12003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "770e83e4819e49fabebce41221489ed1"
format_stamp: "Formatted at 2026-09-29 02:49:48 on dist-test-slave-t6c1"
server_key: "386d137e853f8d9e6696fd342aba2b0e"
server_key_iv: "845e29f216de3d3342542ffac1e1b8f6"
server_key_version: "encryptionkey@0"
I20260929 02:49:51.943408 12003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:51.961751 12003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:51.965863 12003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:51.966069 12003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:51.966961 12003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:51.968319 12003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:51.968405 12003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:51.968510 12003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:51.968586 12003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:51.995340 12003 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:39001
I20260929 02:49:51.995406 12139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:39001 every 8 connection(s)
I20260929 02:49:51.996685 12003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:52.005621 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 12003
I20260929 02:49:52.005939 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 11354
Sep 29 02:49:52 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.131@KRBTEST.COM
Sep 29 02:49:52 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650191, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.188@KRBTEST.COM
Sep 29 02:49:52 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650191, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.189@KRBTEST.COM
Sep 29 02:49:52 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650191, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:52.044839 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:51.999680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:42423 (local address 127.4.197.188:44989)
0929 02:49:52.000225 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.000232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.003203 (+ 2971us) server_negotiation.cc:407] Connection header received
0929 02:49:52.004797 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.004804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.004948 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.005053 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.025530 (+ 20477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.026809 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.029741 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.030461 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.030562 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.037682 (+ 7120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.037689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.037694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.037736 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.040760 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.041716 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.041722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.041725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.041871 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.042421 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.042427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.042430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.042774 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.042899 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.043963 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.044173 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":203,"threads_started":1}
I20260929 02:49:52.045984 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.000228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:44989 (local address 127.4.197.130:42423)
0929 02:49:52.001105 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.001165 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.004317 (+ 3152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.023012 (+ 18695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.023031 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.024024 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:52.024994 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.025023 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.027170 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.027177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.029547 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.029564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.031303 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.031314 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.031664 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.033573 (+ 1909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:52.033603 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:52.037414 (+ 3811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:52.040917 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.040927 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.040945 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.041576 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.042011 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.042015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.042019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.042249 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.043043 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:52.043051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:52.043546 (+ 495us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.043846 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.044447 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":320,"threads_started":1}
I20260929 02:49:52.047758 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.007427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.4.197.131:40099)
0929 02:49:52.022808 (+ 15381us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.022827 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.023433 (+ 606us) server_negotiation.cc:407] Connection header received
0929 02:49:52.023517 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.023527 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.023850 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.024009 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.026363 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.028074 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.030274 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.031223 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.031344 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.037784 (+ 6440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.037802 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.037820 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.037862 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.042870 (+ 5008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.044558 (+ 1688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.044567 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.044571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.044700 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.045144 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.045149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.045152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.045606 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.045892 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.047109 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.047373 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15180,"thread_start_us":17768,"threads_started":1}
I20260929 02:49:52.047977 12140 heartbeater.cc:347] Connected to a master server at 127.4.197.188:44989
I20260929 02:49:52.048307 12140 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:52.049080 11477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:52.049165 12140 heartbeater.cc:511] Master 127.4.197.188:44989 requested a full tablet report, sending...
I20260929 02:49:52.050225 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.000640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:43749 (local address 127.4.197.130:42715)
0929 02:49:52.021740 (+ 21100us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.021764 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.021953 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.032868 (+ 10915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.032874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.033459 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:52.033755 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.033766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.035325 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.035330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.037037 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.037052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.037955 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.037961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.038119 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.038839 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:52.038866 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:52.041831 (+ 2965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:52.047336 (+ 5505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.047341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.047344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.047886 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.048314 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.048319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.048322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.048466 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.049144 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:52.049149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:52.049309 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.049456 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.049765 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20937,"mutex_wait_us":25,"thread_start_us":31812,"threads_started":1}
I20260929 02:49:52.050761 11813 ts_manager.cc:194] Registered new tserver with Master: 770e83e4819e49fabebce41221489ed1 (127.4.197.130:39001)
I20260929 02:49:52.051277 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.000732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:42715 (local address 127.4.197.189:43749)
0929 02:49:52.032511 (+ 31779us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.032517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.032537 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:49:52.032597 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.032602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.032686 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.033921 (+ 1235us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.034111 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.035166 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.037198 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.037757 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.038402 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.042037 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.042043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.042048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.042095 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.047187 (+ 5092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.048035 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.048040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.048044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.048186 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.048589 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.048593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.048596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.048926 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.049071 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.050637 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.050891 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31655,"thread_start_us":31505,"threads_started":1}
I20260929 02:49:52.052065 12142 heartbeater.cc:347] Connected to a master server at 127.4.197.189:43749
I20260929 02:49:52.052175 12142 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:52.052575 12142 heartbeater.cc:511] Master 127.4.197.189:43749 requested a full tablet report, sending...
I20260929 02:49:52.053858 11624 ts_manager.cc:194] Registered new tserver with Master: 770e83e4819e49fabebce41221489ed1 (127.4.197.130:39001)
I20260929 02:49:52.055359 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.001705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.130:37897)
0929 02:49:52.032822 (+ 31117us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.032840 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.032948 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.033298 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.033302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.035765 (+ 2463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:52.036094 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.036104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.038562 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.038565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.040123 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.040136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.041208 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.041215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.041416 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.041963 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:52.041984 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:52.045899 (+ 3915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:52.051154 (+ 5255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.051159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.051162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.052119 (+ 957us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.053367 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.053372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.053376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.053505 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.054013 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:52.054018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:52.054160 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.054351 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.054695 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30988,"mutex_wait_us":25,"thread_start_us":26505,"threads_started":1}
I20260929 02:49:52.055719 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.002065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:37897 (local address 127.4.197.190:38301)
0929 02:49:52.021100 (+ 19035us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.021108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.032923 (+ 11815us) server_negotiation.cc:407] Connection header received
0929 02:49:52.033076 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.033081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.033164 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.033267 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.037253 (+ 3986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.038247 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.040258 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.040916 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.041022 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.046105 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.046112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.046117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.046157 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.049430 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.053105 (+ 3675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.053109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.053111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.053256 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.053614 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.053619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.053623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.053881 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.054060 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.054484 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.054695 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18862,"thread_start_us":19055,"threads_started":1}
I20260929 02:49:52.056465 12141 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38301
I20260929 02:49:52.056581 12141 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:52.056919 12141 heartbeater.cc:511] Master 127.4.197.190:38301 requested a full tablet report, sending...
I20260929 02:49:52.057770 11026 ts_manager.cc:194] Re-registered known tserver with Master: 770e83e4819e49fabebce41221489ed1 (127.4.197.130:39001)
I20260929 02:49:52.058655 11026 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:37897
I20260929 02:49:52.147192 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 11354
I20260929 02:49:52.159837 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:40099
--local_ip_for_outbound_sockets=127.4.197.131
--tserver_master_addrs=127.4.197.190:38301
--webserver_port=44139
--webserver_interface=127.4.197.131
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab
--principal=oryx/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.197.190:38301,127.4.197.189:43749
--tserver_master_addrs=127.4.197.190:38301,127.4.197.189:43749,127.4.197.188:44989 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:49:52.463388 11993 heartbeater.cc:503] Master 127.4.197.190:38301 was elected leader, sending a full tablet report...
W20260929 02:49:52.475322 12153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:52.475603 12153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:52.475739 12153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:52.483275 12153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:52.483407 12153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:52.483556 12153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:52.493675 12153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab
--principal=oryx/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:40099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=44139
--tserver_master_addrs=127.4.197.190:38301,127.4.197.189:43749,127.4.197.188:44989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:52.495204 12153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:52.496658 12153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:52.510419 12159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:52.510464 12161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:52.510490 12158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:52.511176 12153 server_base.cc:1034] running on GCE node
I20260929 02:49:52.511786 12153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:52.512750 12153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:52.513937 12153 hybrid_clock.cc:648] HybridClock initialized: now 1790650192513885 us; error 44 us; skew 500 ppm
I20260929 02:49:52.514227 12153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:52 dist-test-slave-t6c1 krb5kdc[10981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650192, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:52.517303 12153 init.cc:377] Logged in from keytab as oryx/127.4.197.131@KRBTEST.COM (short username oryx)
I20260929 02:49:52.518530 12153 webserver.cc:460] Webserver started at http://127.4.197.131:44139/ using document root <none> and password file <none>
I20260929 02:49:52.519198 12153 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:52.519310 12153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:52.524433 12153 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:52.527158 12168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:52.528517 12153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:52.528668 12153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "bbc41f9a5e014994be48f46fab7d976e"
format_stamp: "Formatted at 2026-09-29 02:49:49 on dist-test-slave-t6c1"
server_key: "9c1da911584ba017b9bffad05004d6fc"
server_key_iv: "0e934a37d57f0c099f26eda5d491881d"
server_key_version: "encryptionkey@0"
I20260929 02:49:52.529654 12153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:52.562261 12153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:52.567127 12153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:52.567481 12153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:52.568583 12153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:52.569895 12153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:52.569988 12153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:52.570087 12153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:52.570163 12153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:52.599148 12153 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:40099
I20260929 02:49:52.599222 12280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:40099 every 8 connection(s)
I20260929 02:49:52.600682 12153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:52.607897 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 12153
Sep 29 02:49:52 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650192, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.188@KRBTEST.COM
Sep 29 02:49:52 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650192, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.189@KRBTEST.COM
I20260929 02:49:52.628273 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.603721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:35165 (local address 127.4.197.188:44989)
0929 02:49:52.604146 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.604151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.605413 (+ 1262us) server_negotiation.cc:407] Connection header received
0929 02:49:52.608478 (+ 3065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.608484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.608575 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.608654 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.611050 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.612234 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.614723 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.615375 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.615470 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.620106 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.620114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.620119 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.620162 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.623797 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.625138 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.625145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.625149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.625316 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.625890 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.625895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.625899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.626528 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.626660 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.627709 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.627926 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":207,"threads_started":1}
Sep 29 02:49:52 dist-test-slave-t6c1 krb5kdc[10981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650192, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:52.630774 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.604395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:44989 (local address 127.4.197.131:35165)
0929 02:49:52.605194 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.605255 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.607977 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.609142 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.609155 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.609980 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:52.610825 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.610849 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.612452 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.612458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.614548 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.614563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.615814 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.615821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.616140 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.616929 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:52.616963 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:52.619827 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:52.623983 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.623993 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.624012 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.624989 (+ 977us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.625467 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.625472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.625476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.625761 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.626775 (+ 1014us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:52.626784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:52.627267 (+ 483us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.628811 (+ 1544us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.629343 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"spinlock_wait_cycles":11776,"thread_start_us":639,"threads_started":1}
I20260929 02:49:52.632611 12281 heartbeater.cc:347] Connected to a master server at 127.4.197.188:44989
I20260929 02:49:52.632931 12281 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:52.633865 12281 heartbeater.cc:511] Master 127.4.197.188:44989 requested a full tablet report, sending...
I20260929 02:49:52.633924 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.604974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:43749 (local address 127.4.197.131:37499)
0929 02:49:52.605539 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.605554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.608235 (+ 2681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.619401 (+ 11166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.619409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.620081 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:52.620341 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.620353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.621598 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.621603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.623020 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.623034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.624115 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.624121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.624301 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.624902 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:52.624922 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:52.627384 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:52.631094 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.631099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.631103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.631641 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.632030 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.632035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.632038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.632181 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.632881 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:52.632886 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:52.633039 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.633167 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.633448 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"spinlock_wait_cycles":16128,"thread_start_us":1385,"threads_started":1}
I20260929 02:49:52.634801 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.605065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:37499 (local address 127.4.197.189:43749)
0929 02:49:52.619116 (+ 14051us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.619120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.619136 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:49:52.619182 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.619185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.619256 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.619352 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.620463 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.621485 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.623169 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.623777 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.623890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.627594 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.627601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.627606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.627654 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.630939 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.631771 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.631776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.631780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.631932 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.632314 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.632320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.632324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.632689 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.632822 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.634197 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.634418 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13957,"thread_start_us":13852,"threads_started":1}
I20260929 02:49:52.636099 12283 heartbeater.cc:347] Connected to a master server at 127.4.197.189:43749
I20260929 02:49:52.636157 11813 ts_manager.cc:194] Registered new tserver with Master: bbc41f9a5e014994be48f46fab7d976e (127.4.197.131:40099)
I20260929 02:49:52.636230 12283 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:52.636711 12283 heartbeater.cc:511] Master 127.4.197.189:43749 requested a full tablet report, sending...
I20260929 02:49:52.636761 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.606246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.131:54409)
0929 02:49:52.616457 (+ 10211us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.616472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.616576 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.620506 (+ 3930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.620511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.621010 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:52.621277 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.621287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.623215 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.623220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.624581 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.624592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.625665 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.625671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.625867 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.626379 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:52.626401 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:52.629558 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:52.633442 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.633447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.633450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.633925 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.634494 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:52.634499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:52.634503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:52.634653 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:52.635845 (+ 1192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:52.635848 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:52.635988 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.636125 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.636381 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10110,"thread_start_us":9804,"threads_started":1}
I20260929 02:49:52.636933 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.606390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:54409 (local address 127.4.197.190:38301)
0929 02:49:52.620088 (+ 13698us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.620093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.620112 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:49:52.620167 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.620172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.620240 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.620341 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.621683 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.622759 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.624741 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.625318 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.625445 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.629754 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.629761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.629765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.629804 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.633089 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.634168 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.634190 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.634194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.634335 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.634776 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.634780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.634783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.635652 (+ 869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.635777 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.636244 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.636463 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13576,"thread_start_us":13484,"threads_started":1}
I20260929 02:49:52.637668 12282 heartbeater.cc:347] Connected to a master server at 127.4.197.190:38301
I20260929 02:49:52.637782 12282 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:52.638032 11624 ts_manager.cc:194] Registered new tserver with Master: bbc41f9a5e014994be48f46fab7d976e (127.4.197.131:40099)
I20260929 02:49:52.638116 12282 heartbeater.cc:511] Master 127.4.197.190:38301 requested a full tablet report, sending...
I20260929 02:49:52.639117 11026 ts_manager.cc:194] Re-registered known tserver with Master: bbc41f9a5e014994be48f46fab7d976e (127.4.197.131:40099)
I20260929 02:49:52.640192 11026 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:54409
I20260929 02:49:52.655493 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.639857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.4.197.188:44989)
0929 02:49:52.640010 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.640016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.641146 (+ 1130us) server_negotiation.cc:407] Connection header received
0929 02:49:52.641287 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.641292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.641363 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.641460 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.643625 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.644394 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.646458 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.647053 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.647150 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.650132 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.650139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.650144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.650185 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.652611 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.653630 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.653634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.653637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.653758 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.654103 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.654109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.654113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.654398 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.654604 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.654943 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.655161 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:49:52.658030 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.629267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.4.197.190:38301)
0929 02:49:52.637158 (+ 7891us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.637164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.640699 (+ 3535us) server_negotiation.cc:407] Connection header received
0929 02:49:52.640758 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.640764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.640862 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.640956 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.643109 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.644264 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.646447 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.647013 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.647134 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.651098 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.651104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.651109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.651153 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.654489 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.655405 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.655411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.655415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.655559 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.655973 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.655978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.655981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.656320 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.656571 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.657459 (+ 888us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.657683 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7788,"thread_start_us":9781,"threads_started":1}
I20260929 02:49:52.658025 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.638995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.4.197.189:43749)
0929 02:49:52.639261 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.639266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.640696 (+ 1430us) server_negotiation.cc:407] Connection header received
0929 02:49:52.643230 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.643235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.643314 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.643412 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.645826 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.646594 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.648474 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.649128 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.649257 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.652194 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.652200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.652205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.652245 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.655479 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.656232 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.656237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.656240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.656390 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.656805 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.656810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.656813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.657057 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.657302 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.657612 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.657770 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 02:49:52.661783 11026 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:52.668934 11026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:52.698444 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.687817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:33895 (local address 127.0.0.1:58244)
0929 02:49:52.688886 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.688903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.689015 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.690095 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.690099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.690122 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:52.690497 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.690509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.692757 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.692762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.695539 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.695556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.697599 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.697605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.697776 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.697809 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.697897 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.698033 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":1241,"threads_started":1}
I20260929 02:49:52.699404 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.688904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.4.197.129:33895)
0929 02:49:52.689425 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.689434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.689460 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:49:52.689561 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.689568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.689878 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.690704 (+ 826us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:52.690973 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.692600 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.695726 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.697129 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.697256 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.697490 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.698692 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.698903 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":546,"threads_started":1}
I20260929 02:49:52.700088 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.689342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:40099 (local address 127.0.0.1:36444)
0929 02:49:52.691318 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.691332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.691439 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.691975 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.691980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.692004 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:52.692351 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.692361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.695032 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.695037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.696943 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.696953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.699235 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.699241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.699408 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.699438 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.699530 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.699681 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1871,"mutex_wait_us":3,"thread_start_us":1751,"threads_started":1}
I20260929 02:49:52.700393 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.689596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.4.197.131:40099)
0929 02:49:52.690667 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.690677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.691426 (+ 749us) server_negotiation.cc:407] Connection header received
0929 02:49:52.691568 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.691576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.691813 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.691980 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:52.692517 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.693958 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.697088 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.699063 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.699179 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.699456 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.699570 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.700086 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":462,"threads_started":1}
I20260929 02:49:52.703263 11927 tablet_service.cc:1467] Processing CreateTablet for tablet 458d26bf6a0c4423961183f99fec69f2 (DEFAULT_TABLE table=test-table [id=1d58f8f97db8457b831c4dd58c55179f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:52.703904 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.687358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:39001 (local address 127.0.0.1:47512)
0929 02:49:52.688413 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.688435 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.688576 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.695864 (+ 7288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.695869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.695895 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:52.696310 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.696322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.698068 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.698073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.700119 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.700132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.703106 (+ 2974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.703112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.703311 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.703342 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.703430 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.703552 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":22,"thread_start_us":595,"threads_started":1}
I20260929 02:49:52.703737 12216 tablet_service.cc:1467] Processing CreateTablet for tablet 458d26bf6a0c4423961183f99fec69f2 (DEFAULT_TABLE table=test-table [id=1d58f8f97db8457b831c4dd58c55179f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:52.704728 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.687832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.4.197.130:39001)
0929 02:49:52.695321 (+ 7489us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.695329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.695357 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:49:52.695440 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.695445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.695655 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.695890 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:52.696509 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.697935 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.700280 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.701320 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.701442 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.701628 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.704315 (+ 2687us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.704477 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7350,"thread_start_us":7260,"threads_started":1}
I20260929 02:49:52.704758 11927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458d26bf6a0c4423961183f99fec69f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:52.704973 12216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458d26bf6a0c4423961183f99fec69f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:52.708117 12075 tablet_service.cc:1467] Processing CreateTablet for tablet 458d26bf6a0c4423961183f99fec69f2 (DEFAULT_TABLE table=test-table [id=1d58f8f97db8457b831c4dd58c55179f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:52.709914 12075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458d26bf6a0c4423961183f99fec69f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:52.716460 12312 tablet_bootstrap.cc:492] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: Bootstrap starting.
I20260929 02:49:52.719632 12313 tablet_bootstrap.cc:492] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: Bootstrap starting.
I20260929 02:49:52.720010 12312 tablet_bootstrap.cc:654] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:52.721309 12312 log.cc:824] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:52.723120 12313 tablet_bootstrap.cc:654] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:52.724265 12313 log.cc:824] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:52.724366 12312 tablet_bootstrap.cc:492] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: No bootstrap required, opened a new log
I20260929 02:49:52.724659 12312 ts_tablet_manager.cc:1397] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 02:49:52.725169 12315 tablet_bootstrap.cc:492] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: Bootstrap starting.
I20260929 02:49:52.728374 12315 tablet_bootstrap.cc:654] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:52.729355 12315 log.cc:824] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:52.731974 12315 tablet_bootstrap.cc:492] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: No bootstrap required, opened a new log
I20260929 02:49:52.732223 12315 ts_tablet_manager.cc:1397] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 02:49:52.737335 12312 raft_consensus.cc:348] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } } peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } }
I20260929 02:49:52.737766 12313 tablet_bootstrap.cc:492] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: No bootstrap required, opened a new log
I20260929 02:49:52.737977 12312 raft_consensus.cc:374] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:52.738095 12313 ts_tablet_manager.cc:1397] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.013s
I20260929 02:49:52.738126 12312 raft_consensus.cc:729] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8df5fbcb6e46e09832e71a7ef19da7, State: Initialized, Role: FOLLOWER
I20260929 02:49:52.738955 12312 consensus_queue.cc:260] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } } peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } }
I20260929 02:49:52.740746 12312 ts_tablet_manager.cc:1428] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260929 02:49:52.741070 12315 raft_consensus.cc:348] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } } peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } }
I20260929 02:49:52.741622 12315 raft_consensus.cc:374] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:52.741781 12315 raft_consensus.cc:729] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770e83e4819e49fabebce41221489ed1, State: Initialized, Role: FOLLOWER
I20260929 02:49:52.743198 12315 consensus_queue.cc:260] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } } peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } }
I20260929 02:49:52.747541 12141 heartbeater.cc:503] Master 127.4.197.190:38301 was elected leader, sending a full tablet report...
I20260929 02:49:52.748018 12313 raft_consensus.cc:348] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } } peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } }
I20260929 02:49:52.748548 12313 raft_consensus.cc:374] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:52.748653 12313 raft_consensus.cc:729] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc41f9a5e014994be48f46fab7d976e, State: Initialized, Role: FOLLOWER
I20260929 02:49:52.749531 12315 ts_tablet_manager.cc:1428] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260929 02:49:52.749467 12313 consensus_queue.cc:260] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } } peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } }
W20260929 02:49:52.750082 12143 tablet.cc:2369] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:52.750661 12282 heartbeater.cc:503] Master 127.4.197.190:38301 was elected leader, sending a full tablet report...
I20260929 02:49:52.751328 12313 ts_tablet_manager.cc:1428] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20260929 02:49:52.763581 12320 raft_consensus.cc:482] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:52.763865 12320 raft_consensus.cc:504] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } } peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } }
I20260929 02:49:52.765401 12320 leader_election.cc:288] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 770e83e4819e49fabebce41221489ed1 (127.4.197.130:39001), af8df5fbcb6e46e09832e71a7ef19da7 (127.4.197.129:33895)
I20260929 02:49:52.771598 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.766135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:45201 (local address 127.4.197.130:39001)
0929 02:49:52.766301 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.766306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.766326 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:49:52.766396 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.766401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.766481 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.766568 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:52.767002 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.768016 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.769803 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.770628 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.770741 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.770802 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.771211 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.771339 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:49:52.771597 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.765932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:55549 (local address 127.4.197.129:33895)
0929 02:49:52.766095 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.766100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.766497 (+ 397us) server_negotiation.cc:407] Connection header received
0929 02:49:52.766646 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.766651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.766720 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.766837 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:52.767207 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.768197 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.769946 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.770705 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.770858 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.770951 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.771255 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.771378 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:49:52.771610 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.765606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:39001 (local address 127.4.197.131:45201)
0929 02:49:52.766136 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.766156 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.766281 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.766591 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.766595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.766629 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:52.766812 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.766824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.768161 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.768166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.769651 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.769663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.770806 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.770812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.771060 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.771093 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.771167 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.771302 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":24}
I20260929 02:49:52.771633 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.766243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:33895 (local address 127.4.197.131:55549)
0929 02:49:52.766402 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:52.766418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:52.766526 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:52.766861 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:52.766865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:52.766888 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:52.767071 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.767082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.768345 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.768350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.769809 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:52.769822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.770860 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.770867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:52.771110 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.771133 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:49:52.771219 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 02:49:52.771363 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":3}
I20260929 02:49:52.772601 11947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458d26bf6a0c4423961183f99fec69f2" candidate_uuid: "bbc41f9a5e014994be48f46fab7d976e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" is_pre_election: true
I20260929 02:49:52.772625 12095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458d26bf6a0c4423961183f99fec69f2" candidate_uuid: "bbc41f9a5e014994be48f46fab7d976e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770e83e4819e49fabebce41221489ed1" is_pre_election: true
I20260929 02:49:52.773043 12095 raft_consensus.cc:2454] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbc41f9a5e014994be48f46fab7d976e in term 0.
I20260929 02:49:52.773038 11947 raft_consensus.cc:2454] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbc41f9a5e014994be48f46fab7d976e in term 0.
I20260929 02:49:52.773622 12170 leader_election.cc:302] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 770e83e4819e49fabebce41221489ed1, bbc41f9a5e014994be48f46fab7d976e; no voters:
I20260929 02:49:52.773985 12320 raft_consensus.cc:2781] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:52.774118 12320 raft_consensus.cc:482] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:52.774215 12320 raft_consensus.cc:3037] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:52.776667 12320 raft_consensus.cc:504] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } } peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } }
I20260929 02:49:52.777240 12320 leader_election.cc:288] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [CANDIDATE]: Term 1 election: Requested vote from peers 770e83e4819e49fabebce41221489ed1 (127.4.197.130:39001), af8df5fbcb6e46e09832e71a7ef19da7 (127.4.197.129:33895)
I20260929 02:49:52.777740 12095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458d26bf6a0c4423961183f99fec69f2" candidate_uuid: "bbc41f9a5e014994be48f46fab7d976e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770e83e4819e49fabebce41221489ed1"
I20260929 02:49:52.777817 11947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458d26bf6a0c4423961183f99fec69f2" candidate_uuid: "bbc41f9a5e014994be48f46fab7d976e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8df5fbcb6e46e09832e71a7ef19da7"
I20260929 02:49:52.777915 12095 raft_consensus.cc:3037] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:52.778048 11947 raft_consensus.cc:3037] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:52.780252 12095 raft_consensus.cc:2454] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbc41f9a5e014994be48f46fab7d976e in term 1.
I20260929 02:49:52.780611 12170 leader_election.cc:302] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 770e83e4819e49fabebce41221489ed1, bbc41f9a5e014994be48f46fab7d976e; no voters:
I20260929 02:49:52.780869 12320 raft_consensus.cc:2781] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:52.780956 11947 raft_consensus.cc:2454] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbc41f9a5e014994be48f46fab7d976e in term 1.
I20260929 02:49:52.781304 12320 raft_consensus.cc:686] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 1 LEADER]: Becoming Leader. State: Replica: bbc41f9a5e014994be48f46fab7d976e, State: Running, Role: LEADER
I20260929 02:49:52.781828 12320 consensus_queue.cc:237] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } } peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } }
I20260929 02:49:52.786566 11025 catalog_manager.cc:5495] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e reported cstate change: term changed from 0 to 1, leader changed from <none> to bbc41f9a5e014994be48f46fab7d976e (127.4.197.131). New cstate: current_term: 1 leader_uuid: "bbc41f9a5e014994be48f46fab7d976e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } health_report { overall_health: HEALTHY } } }
I20260929 02:49:52.823796 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.819148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.4.197.131:40099)
0929 02:49:52.819357 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.819364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.819462 (+ 98us) server_negotiation.cc:407] Connection header received
0929 02:49:52.819624 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.819628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.819701 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.819821 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:52.820129 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.820779 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.821967 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.822331 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.822424 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.822787 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:52.823318 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:52.823420 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.823493 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.823554 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:49:52.831703 11947 raft_consensus.cc:1261] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7 [term 1 FOLLOWER]: Refusing update from remote peer bbc41f9a5e014994be48f46fab7d976e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:52.831707 12095 raft_consensus.cc:1261] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1 [term 1 FOLLOWER]: Refusing update from remote peer bbc41f9a5e014994be48f46fab7d976e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:52.832661 12321 consensus_queue.cc:1035] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [LEADER]: Connected to new peer: Peer: permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:52.832999 12320 consensus_queue.cc:1035] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:52.846221 12326 mvcc.cc:205] Tried to move back new op lower bound from 7334503189828870144 to 7334503189638111232. Current Snapshot: MvccSnapshot[applied={T|T < 7334503189828870144}]
I20260929 02:49:52.846722 12325 mvcc.cc:205] Tried to move back new op lower bound from 7334503189828870144 to 7334503189638111232. Current Snapshot: MvccSnapshot[applied={T|T < 7334503189828870144}]
I20260929 02:49:52.868332 11026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:52.868628 11026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:52.875352 11026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ae4591dcf7b454cbaed46d286281fc8: Sending DeleteTablet for 3 replicas of tablet 458d26bf6a0c4423961183f99fec69f2
I20260929 02:49:52.876706 12075 tablet_service.cc:1514] Processing DeleteTablet for tablet 458d26bf6a0c4423961183f99fec69f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:52 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:47512
I20260929 02:49:52.876982 11927 tablet_service.cc:1514] Processing DeleteTablet for tablet 458d26bf6a0c4423961183f99fec69f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:52 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:58244
I20260929 02:49:52.877902 12216 tablet_service.cc:1514] Processing DeleteTablet for tablet 458d26bf6a0c4423961183f99fec69f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:52 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:36444
I20260929 02:49:52.878520 12336 tablet_replica.cc:330] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: stopping tablet replica
I20260929 02:49:52.878734 12335 tablet_replica.cc:330] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: stopping tablet replica
I20260929 02:49:52.879009 12336 raft_consensus.cc:2229] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:52.879120 12335 raft_consensus.cc:2229] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:52.879511 12335 raft_consensus.cc:2258] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:52.879513 12336 raft_consensus.cc:2258] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:52.881793 12336 ts_tablet_manager.cc:1905] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:52.881781 12335 ts_tablet_manager.cc:1905] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:52.888742 12335 ts_tablet_manager.cc:1918] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:52.889158 12336 ts_tablet_manager.cc:1918] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:52.889379 12336 log.cc:1196] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/458d26bf6a0c4423961183f99fec69f2
I20260929 02:49:52.890110 12337 tablet_replica.cc:330] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: stopping tablet replica
I20260929 02:49:52.890570 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 10994
I20260929 02:49:52.890679 12337 raft_consensus.cc:2229] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:52.891472 12337 raft_consensus.cc:2258] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:52.892431 11056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:52.893958 12337 ts_tablet_manager.cc:1905] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:52.897727 12335 log.cc:1196] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/458d26bf6a0c4423961183f99fec69f2
I20260929 02:49:52.898427 12335 ts_tablet_manager.cc:1939] T 458d26bf6a0c4423961183f99fec69f2 P 770e83e4819e49fabebce41221489ed1: Deleting consensus metadata
I20260929 02:49:52.892125 12336 ts_tablet_manager.cc:1939] T 458d26bf6a0c4423961183f99fec69f2 P af8df5fbcb6e46e09832e71a7ef19da7: Deleting consensus metadata
I20260929 02:49:52.901708 12337 ts_tablet_manager.cc:1918] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:52.901935 12337 log.cc:1196] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/458d26bf6a0c4423961183f99fec69f2
I20260929 02:49:52.902565 12337 ts_tablet_manager.cc:1939] T 458d26bf6a0c4423961183f99fec69f2 P bbc41f9a5e014994be48f46fab7d976e: Deleting consensus metadata
I20260929 02:49:52.908957 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.879651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.4.197.189:43749)
0929 02:49:52.879829 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.879835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.889553 (+ 9718us) server_negotiation.cc:407] Connection header received
0929 02:49:52.889849 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.889855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.889941 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.890014 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.892329 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.893495 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.895543 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.896199 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.896308 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.902776 (+ 6468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.902783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.902789 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.902837 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.906202 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.907056 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.907061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.907065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.907210 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.907574 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.907579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.907583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.907885 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.908131 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.908428 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.908630 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:49:52.921243 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:52.889671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.4.197.188:44989)
0929 02:49:52.889853 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:52.889858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:52.909157 (+ 19299us) server_negotiation.cc:407] Connection header received
0929 02:49:52.909388 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:52.909393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:52.909476 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:52.909544 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:52.911237 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.911943 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.913174 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:52.913688 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:52.913794 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:52.916291 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:52.916297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:52.916303 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:52.916345 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:52.918602 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.919274 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.919280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.919282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.919439 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:52.919892 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:52.919897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:52.919901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:52.920154 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:52.920409 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:52.920721 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 02:49:52.920921 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:49:52.904850 11011 catalog_manager.cc:4841] TS 770e83e4819e49fabebce41221489ed1 (127.4.197.130:39001): tablet 458d26bf6a0c4423961183f99fec69f2 (table test-table [id=1d58f8f97db8457b831c4dd58c55179f]) successfully deleted
W20260929 02:49:53.047832 11011 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.197.130:39001, user_credentials={real_user=slave}} blocked reactor thread for 142954us
I20260929 02:49:53.048069 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 10994
W20260929 02:49:53.065609 12170 connection.cc:535] server connection from 127.0.0.1:36444 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:49:53.093541 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:53.074643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.4.197.188:44989)
0929 02:49:53.074841 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:53.074848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:53.075840 (+ 992us) server_negotiation.cc:407] Connection header received
0929 02:49:53.076140 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:53.076146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:53.076228 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:53.076300 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:53.078701 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:53.079863 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:53.081871 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:53.083094 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:53.083244 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:53.086840 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:53.086848 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:53.086853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:53.086904 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:53.090491 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:53.091440 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:53.091446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:53.091450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:53.091598 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:53.091999 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:53.092004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:53.092007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:53.092354 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:53.092513 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:53.093003 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 02:49:53.093209 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:49:53.094583 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:53.073250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.4.197.189:43749)
0929 02:49:53.073456 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:53.073462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:53.074962 (+ 1500us) server_negotiation.cc:407] Connection header received
0929 02:49:53.075241 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:53.075248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:53.075334 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:53.075406 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:53.077576 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:53.078670 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:53.080660 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:53.081344 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:53.081435 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:53.085123 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:53.085131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:53.085136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:53.085178 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:53.089022 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:53.091546 (+ 2524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:53.091553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:53.091558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:53.091713 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:53.092176 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:53.092182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:53.092186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:53.092554 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:53.092780 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:53.093462 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 02:49:53.094040 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:49:53.855829 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:53.855162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.129:50873)
0929 02:49:53.855581 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:53.855680 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":222,"threads_started":1}
W20260929 02:49:53.856235 11993 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:38301 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
I20260929 02:49:53.859365 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:53.858538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.131:59593)
0929 02:49:53.858989 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:53.859136 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":213,"threads_started":1}
W20260929 02:49:53.859774 12282 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:38301 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
I20260929 02:49:53.860041 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:53.859382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.130:48647)
0929 02:49:53.859778 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:53.859900 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":364,"threads_started":1}
W20260929 02:49:53.860380 12141 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:38301 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
I20260929 02:49:54.375671 12349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ae4591dcf7b454cbaed46d286281fc8)
I20260929 02:49:54.375833 12349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } }
I20260929 02:49:54.376453 12350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ae4591dcf7b454cbaed46d286281fc8)
I20260929 02:49:54.376535 12349 leader_election.cc:288] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ae4591dcf7b454cbaed46d286281fc8 (127.4.197.190:38301), 33894a7c546c4f3eaebaeaa10dc84b6e (127.4.197.189:43749)
I20260929 02:49:54.376850 12350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } }
I20260929 02:49:54.378566 12350 leader_election.cc:288] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ae4591dcf7b454cbaed46d286281fc8 (127.4.197.190:38301), 4618292714304d2190f0fb80c1582564 (127.4.197.188:44989)
I20260929 02:49:54.379717 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.376667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40718)
0929 02:49:54.379215 (+ 2548us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.379440 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2404,"threads_started":1}
W20260929 02:49:54.380057 11798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
I20260929 02:49:54.380564 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.379458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40734)
0929 02:49:54.380110 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.380315 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":411,"threads_started":1}
W20260929 02:49:54.381575 11609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
I20260929 02:49:54.383373 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.382899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40736)
0929 02:49:54.383118 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.383202 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260929 02:49:54.383610 11798 leader_election.cc:334] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ae4591dcf7b454cbaed46d286281fc8 (127.4.197.190:38301): Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
I20260929 02:49:54.384220 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.383658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40742)
0929 02:49:54.383875 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.384028 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260929 02:49:54.384423 11609 leader_election.cc:334] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ae4591dcf7b454cbaed46d286281fc8 (127.4.197.190:38301): Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
I20260929 02:49:54.387323 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.376881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.4.197.189:43749)
0929 02:49:54.379450 (+ 2569us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:54.379456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:54.379952 (+ 496us) server_negotiation.cc:407] Connection header received
0929 02:49:54.380223 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:54.380229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:54.380324 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:54.380400 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:54.381244 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.382571 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.385445 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.386365 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.386474 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.386561 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:54.386833 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 02:49:54.386986 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":2452,"thread_start_us":2279,"threads_started":1}
I20260929 02:49:54.387372 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.376730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:43749 (local address 127.0.0.1:55918)
0929 02:49:54.379864 (+ 3134us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.379882 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:54.380059 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:54.380648 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:54.380653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:54.380685 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:54.381048 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:54.381061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.383474 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.383480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:54.385297 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:54.385310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.386528 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.386534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:54.386690 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.386713 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:49:54.386797 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 02:49:54.386982 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3012,"mutex_wait_us":1,"thread_start_us":2150,"threads_started":1}
I20260929 02:49:54.387878 11634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4618292714304d2190f0fb80c1582564" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" is_pre_election: true
I20260929 02:49:54.388170 11634 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4618292714304d2190f0fb80c1582564 in term 1.
I20260929 02:49:54.388258 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.380680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:44989 (local address 127.0.0.1:37440)
0929 02:49:54.380876 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.380900 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:54.381018 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:54.382833 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:54.382839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:54.382868 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:54.383253 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:54.383266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.384885 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.384891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:54.386260 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:54.386272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.387547 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.387552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:54.387707 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.387738 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:49:54.387815 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:49:54.387943 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":19,"thread_start_us":555,"threads_started":1}
I20260929 02:49:54.388669 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.380366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.4.197.188:44989)
0929 02:49:54.381885 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:54.381892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:54.381916 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:49:54.381981 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:54.381986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:54.382059 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:54.383404 (+ 1345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:54.383716 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.384756 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.386418 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.387347 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.388154 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.388224 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:54.388262 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 02:49:54.388393 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":1381,"thread_start_us":1417,"threads_started":1}
I20260929 02:49:54.388631 11797 leader_election.cc:302] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33894a7c546c4f3eaebaeaa10dc84b6e, 4618292714304d2190f0fb80c1582564; no voters: 4ae4591dcf7b454cbaed46d286281fc8
I20260929 02:49:54.388932 12349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 02:49:54.389041 12349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4ae4591dcf7b454cbaed46d286281fc8)
I20260929 02:49:54.389132 12349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:49:54.389204 11823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4618292714304d2190f0fb80c1582564" is_pre_election: true
I20260929 02:49:54.391247 12349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } }
I20260929 02:49:54.391659 11823 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 33894a7c546c4f3eaebaeaa10dc84b6e in current term 2: Already voted for candidate 4618292714304d2190f0fb80c1582564 in this term.
I20260929 02:49:54.391891 12349 leader_election.cc:288] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [CANDIDATE]: Term 2 election: Requested vote from peers 4ae4591dcf7b454cbaed46d286281fc8 (127.4.197.190:38301), 33894a7c546c4f3eaebaeaa10dc84b6e (127.4.197.189:43749)
I20260929 02:49:54.392361 11634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4618292714304d2190f0fb80c1582564" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e"
I20260929 02:49:54.392472 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.391961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40744)
0929 02:49:54.392273 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.392356 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260929 02:49:54.392498 11609 leader_election.cc:302] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33894a7c546c4f3eaebaeaa10dc84b6e; no voters: 4618292714304d2190f0fb80c1582564, 4ae4591dcf7b454cbaed46d286281fc8
I20260929 02:49:54.392580 11634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:49:54.393433 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.393130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40748)
0929 02:49:54.393249 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.393331 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 02:49:54.393584 11798 leader_election.cc:334] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4ae4591dcf7b454cbaed46d286281fc8 (127.4.197.190:38301): Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
I20260929 02:49:54.394783 11634 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4618292714304d2190f0fb80c1582564 in term 2.
I20260929 02:49:54.394979 12350 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:49:54.395155 11797 leader_election.cc:302] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33894a7c546c4f3eaebaeaa10dc84b6e, 4618292714304d2190f0fb80c1582564; no voters: 4ae4591dcf7b454cbaed46d286281fc8
I20260929 02:49:54.395445 12349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:49:54.395906 12349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [term 2 LEADER]: Becoming Leader. State: Replica: 4618292714304d2190f0fb80c1582564, State: Running, Role: LEADER
I20260929 02:49:54.396354 12349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } }
I20260929 02:49:54.398157 12359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4618292714304d2190f0fb80c1582564. Latest consensus state: current_term: 2 leader_uuid: "4618292714304d2190f0fb80c1582564" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:54.398550 12359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:54.399283 12361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:54.402994 12361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:54.403815 12361 catalog_manager.cc:1259] Loaded cluster ID: f1b42ab5867d47a98159071347df89ea
I20260929 02:49:54.403892 12361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:54.405051 12361 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:54.405880 12361 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Loaded TSK: 0
I20260929 02:49:54.406776 12361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:49:54.425313 11812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:54.432230 11812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:54.436461 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.436081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40762)
0929 02:49:54.436230 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.436327 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 02:49:54.436640 11634 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [term 2 FOLLOWER]: Refusing update from remote peer 4618292714304d2190f0fb80c1582564: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 02:49:54.437507 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.437102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40764)
0929 02:49:54.437255 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.437334 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:49:54.437451 12359 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 02:49:54.437738 11798 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 -> Peer 4ae4591dcf7b454cbaed46d286281fc8 (127.4.197.190:38301): Couldn't send request to peer 4ae4591dcf7b454cbaed46d286281fc8. Status: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:49:54.440168 12350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4618292714304d2190f0fb80c1582564. Latest consensus state: current_term: 2 leader_uuid: "4618292714304d2190f0fb80c1582564" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:54.440435 12350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:54.441844 12349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4618292714304d2190f0fb80c1582564" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:54.442190 12349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:54.442806 12350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4618292714304d2190f0fb80c1582564" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4ae4591dcf7b454cbaed46d286281fc8" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38301 } } peers { permanent_uuid: "33894a7c546c4f3eaebaeaa10dc84b6e" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "4618292714304d2190f0fb80c1582564" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 44989 } attrs { promote: false } } }
I20260929 02:49:54.443099 12350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33894a7c546c4f3eaebaeaa10dc84b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:54.458427 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.450837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:40099 (local address 127.0.0.1:36464)
0929 02:49:54.451066 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.451087 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:54.451211 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:54.452241 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:54.452247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:54.452330 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:54.452666 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:54.452678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.454328 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.454334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:54.456152 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:54.456163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.457686 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.457693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:54.457861 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.457886 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:49:54.457961 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 02:49:54.458107 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"spinlock_wait_cycles":1408}
I20260929 02:49:54.458407 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.451151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.4.197.131:40099)
0929 02:49:54.451890 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:54.451898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:54.451919 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:49:54.451995 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:54.452000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:54.452078 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:54.452809 (+ 731us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:54.452993 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.454172 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.456312 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.457480 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.457594 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.457701 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:54.458004 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 02:49:54.458133 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":480,"threads_started":1}
I20260929 02:49:54.459486 12216 tablet_service.cc:1467] Processing CreateTablet for tablet 32a7e4401e9e44068ca9030204c8bd62 (DEFAULT_TABLE table=test-table [id=ade9a790128644d2902fa992c3477dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:54.459928 12216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a7e4401e9e44068ca9030204c8bd62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:54.461766 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.451675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:39001 (local address 127.0.0.1:47526)
0929 02:49:54.452185 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.452202 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:54.452307 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:54.455701 (+ 3394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:54.455706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:54.455733 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:54.456070 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:54.456081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.457469 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.457476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:54.459492 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:54.459506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.460986 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.460992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:54.461243 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.461288 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 02:49:54.461366 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:49:54.461474 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":43,"thread_start_us":452,"threads_started":1}
I20260929 02:49:54.462260 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.452051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.4.197.130:39001)
0929 02:49:54.455362 (+ 3311us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:54.455369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:54.455391 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:49:54.455469 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:54.455475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:54.455557 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:54.455669 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:54.456268 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.457327 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.459672 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.460633 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.460745 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.460837 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:54.461941 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0929 02:49:54.462060 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3158,"thread_start_us":3046,"threads_started":1}
I20260929 02:49:54.462949 12075 tablet_service.cc:1467] Processing CreateTablet for tablet 32a7e4401e9e44068ca9030204c8bd62 (DEFAULT_TABLE table=test-table [id=ade9a790128644d2902fa992c3477dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:54.463310 12075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a7e4401e9e44068ca9030204c8bd62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:54.465016 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.450537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:33895 (local address 127.0.0.1:58256)
0929 02:49:54.450699 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.450718 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:54.450820 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:54.459573 (+ 8753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:54.459578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:54.459606 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:54.459755 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:54.459764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.460999 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.461005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:54.462846 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:54.462860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.464339 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.464346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:54.464517 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.464550 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:49:54.464637 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 02:49:54.464768 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":2}
I20260929 02:49:54.465093 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.450645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.4.197.129:33895)
0929 02:49:54.459260 (+ 8615us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:54.459266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:54.459284 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:49:54.459348 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:54.459352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:54.459432 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:54.459534 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:54.459901 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.460843 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.463079 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.464182 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.464303 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.464401 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:54.464673 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 02:49:54.464815 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8500,"thread_start_us":8399,"threads_started":1}
I20260929 02:49:54.465984 11927 tablet_service.cc:1467] Processing CreateTablet for tablet 32a7e4401e9e44068ca9030204c8bd62 (DEFAULT_TABLE table=test-table [id=ade9a790128644d2902fa992c3477dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:54.466274 12372 tablet_bootstrap.cc:492] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e: Bootstrap starting.
I20260929 02:49:54.466421 11927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a7e4401e9e44068ca9030204c8bd62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:54.467785 12372 tablet_bootstrap.cc:654] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:54.469532 12372 tablet_bootstrap.cc:492] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e: No bootstrap required, opened a new log
I20260929 02:49:54.469641 12372 ts_tablet_manager.cc:1397] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:49:54.470242 12372 raft_consensus.cc:348] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } } peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } }
I20260929 02:49:54.470459 12372 raft_consensus.cc:374] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:54.470548 12372 raft_consensus.cc:729] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc41f9a5e014994be48f46fab7d976e, State: Initialized, Role: FOLLOWER
I20260929 02:49:54.470757 12372 consensus_queue.cc:260] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } } peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } }
I20260929 02:49:54.471395 12374 tablet_bootstrap.cc:492] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7: Bootstrap starting.
I20260929 02:49:54.471477 12372 ts_tablet_manager.cc:1428] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:54.473097 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.472181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.131:59989)
0929 02:49:54.472647 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.472837 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":515,"threads_started":1}
I20260929 02:49:54.475181 12281 heartbeater.cc:503] Master 127.4.197.188:44989 was elected leader, sending a full tablet report...
I20260929 02:49:54.475266 12377 tablet_bootstrap.cc:492] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1: Bootstrap starting.
I20260929 02:49:54.476755 12374 tablet_bootstrap.cc:654] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:54.476802 12377 tablet_bootstrap.cc:654] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:54.477062 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.475579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.131:53225)
0929 02:49:54.476877 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.476954 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1197}
I20260929 02:49:54.478767 12374 tablet_bootstrap.cc:492] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7: No bootstrap required, opened a new log
I20260929 02:49:54.478910 12374 ts_tablet_manager.cc:1397] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260929 02:49:54.479288 12377 tablet_bootstrap.cc:492] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1: No bootstrap required, opened a new log
I20260929 02:49:54.479463 12377 ts_tablet_manager.cc:1397] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260929 02:49:54.479724 12374 raft_consensus.cc:348] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } } peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } }
I20260929 02:49:54.479966 12374 raft_consensus.cc:374] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:54.480063 12374 raft_consensus.cc:729] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8df5fbcb6e46e09832e71a7ef19da7, State: Initialized, Role: FOLLOWER
I20260929 02:49:54.480208 12377 raft_consensus.cc:348] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } } peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } }
I20260929 02:49:54.480441 12377 raft_consensus.cc:374] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:54.480317 12374 consensus_queue.cc:260] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } } peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } }
I20260929 02:49:54.480526 12377 raft_consensus.cc:729] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770e83e4819e49fabebce41221489ed1, State: Initialized, Role: FOLLOWER
I20260929 02:49:54.480770 12377 consensus_queue.cc:260] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } } peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } }
I20260929 02:49:54.481106 12374 ts_tablet_manager.cc:1428] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:54.482261 12377 ts_tablet_manager.cc:1428] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:54.482434 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.481357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.129:56445)
0929 02:49:54.481981 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.482155 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":288,"threads_started":1}
I20260929 02:49:54.485579 11992 heartbeater.cc:503] Master 127.4.197.188:44989 was elected leader, sending a full tablet report...
I20260929 02:49:54.486225 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.485152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.129:44125)
0929 02:49:54.485307 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.486075 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 02:49:54.486521 12140 heartbeater.cc:503] Master 127.4.197.188:44989 was elected leader, sending a full tablet report...
I20260929 02:49:54.486845 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.484445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.130:50885)
0929 02:49:54.485017 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.486509 (+ 1492us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":554,"threads_started":1}
I20260929 02:49:54.488747 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.488423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.130:54397)
0929 02:49:54.488566 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.488639 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:49:54.519078 12375 raft_consensus.cc:482] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:54.519289 12375 raft_consensus.cc:504] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } } peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } }
I20260929 02:49:54.519975 12375 leader_election.cc:288] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af8df5fbcb6e46e09832e71a7ef19da7 (127.4.197.129:33895), 770e83e4819e49fabebce41221489ed1 (127.4.197.130:39001)
I20260929 02:49:54.520503 12095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a7e4401e9e44068ca9030204c8bd62" candidate_uuid: "bbc41f9a5e014994be48f46fab7d976e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770e83e4819e49fabebce41221489ed1" is_pre_election: true
I20260929 02:49:54.520502 11947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a7e4401e9e44068ca9030204c8bd62" candidate_uuid: "bbc41f9a5e014994be48f46fab7d976e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" is_pre_election: true
I20260929 02:49:54.520764 11947 raft_consensus.cc:2454] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbc41f9a5e014994be48f46fab7d976e in term 0.
I20260929 02:49:54.520746 12095 raft_consensus.cc:2454] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbc41f9a5e014994be48f46fab7d976e in term 0.
I20260929 02:49:54.521198 12172 leader_election.cc:302] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 770e83e4819e49fabebce41221489ed1, af8df5fbcb6e46e09832e71a7ef19da7, bbc41f9a5e014994be48f46fab7d976e; no voters:
I20260929 02:49:54.521500 12375 raft_consensus.cc:2781] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:54.521605 12375 raft_consensus.cc:482] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:54.521687 12375 raft_consensus.cc:3037] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:54.523729 12375 raft_consensus.cc:504] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } } peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } }
I20260929 02:49:54.524211 12375 leader_election.cc:288] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [CANDIDATE]: Term 1 election: Requested vote from peers af8df5fbcb6e46e09832e71a7ef19da7 (127.4.197.129:33895), 770e83e4819e49fabebce41221489ed1 (127.4.197.130:39001)
I20260929 02:49:54.524698 11947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a7e4401e9e44068ca9030204c8bd62" candidate_uuid: "bbc41f9a5e014994be48f46fab7d976e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8df5fbcb6e46e09832e71a7ef19da7"
I20260929 02:49:54.524698 12095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a7e4401e9e44068ca9030204c8bd62" candidate_uuid: "bbc41f9a5e014994be48f46fab7d976e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770e83e4819e49fabebce41221489ed1"
I20260929 02:49:54.524901 12095 raft_consensus.cc:3037] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:54.524905 11947 raft_consensus.cc:3037] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:54.527727 12095 raft_consensus.cc:2454] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbc41f9a5e014994be48f46fab7d976e in term 1.
I20260929 02:49:54.527727 11947 raft_consensus.cc:2454] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbc41f9a5e014994be48f46fab7d976e in term 1.
I20260929 02:49:54.528083 12170 leader_election.cc:302] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 770e83e4819e49fabebce41221489ed1, bbc41f9a5e014994be48f46fab7d976e; no voters:
I20260929 02:49:54.528335 12375 raft_consensus.cc:2781] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:54.528728 12375 raft_consensus.cc:686] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 1 LEADER]: Becoming Leader. State: Replica: bbc41f9a5e014994be48f46fab7d976e, State: Running, Role: LEADER
I20260929 02:49:54.529157 12375 consensus_queue.cc:237] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } } peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } }
I20260929 02:49:54.529719 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.529238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.131:47923)
0929 02:49:54.529418 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.529527 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 02:49:54.531742 11812 catalog_manager.cc:5495] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e reported cstate change: term changed from 0 to 1, leader changed from <none> to bbc41f9a5e014994be48f46fab7d976e (127.4.197.131). New cstate: current_term: 1 leader_uuid: "bbc41f9a5e014994be48f46fab7d976e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbc41f9a5e014994be48f46fab7d976e" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 } health_report { overall_health: UNKNOWN } } }
I20260929 02:49:54.557799 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.553807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.4.197.131:40099)
0929 02:49:54.553978 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:54.553983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:54.553999 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:49:54.554057 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:54.554060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:54.554124 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:54.554208 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:54.554482 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.555230 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.556409 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.556833 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.556922 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.557270 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:54.557382 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:54.557453 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:54.557542 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 02:49:54.557606 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:49:54.560806 11947 raft_consensus.cc:1261] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7 [term 1 FOLLOWER]: Refusing update from remote peer bbc41f9a5e014994be48f46fab7d976e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:54.560813 12095 raft_consensus.cc:1261] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1 [term 1 FOLLOWER]: Refusing update from remote peer bbc41f9a5e014994be48f46fab7d976e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:54.561400 12384 consensus_queue.cc:1035] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [LEADER]: Connected to new peer: Peer: permanent_uuid: "770e83e4819e49fabebce41221489ed1" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:54.561720 12375 consensus_queue.cc:1035] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8df5fbcb6e46e09832e71a7ef19da7" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:54.564713 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.564256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.129:44987)
0929 02:49:54.564409 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.564526 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 02:49:54.564842 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.563879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.130:50345)
0929 02:49:54.564511 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.564632 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492}
I20260929 02:49:54.567500 12388 mvcc.cc:205] Tried to move back new op lower bound from 7334503196915286016 to 7334503196793597952. Current Snapshot: MvccSnapshot[applied={T|T < 7334503196915286016}]
I20260929 02:49:54.568704 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.568116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.130:43925)
0929 02:49:54.568490 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.568573 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275}
I20260929 02:49:54.569927 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.569316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.131:50205)
0929 02:49:54.569697 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.569780 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260929 02:49:54.574028 11813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:54.574311 11813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:54.581947 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.581527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.129:56721)
0929 02:49:54.581692 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.581787 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 02:49:54.582188 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.581824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.4.197.131:52939)
0929 02:49:54.581948 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.582033 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 02:49:54.583480 11813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4618292714304d2190f0fb80c1582564: Sending DeleteTablet for 3 replicas of tablet 32a7e4401e9e44068ca9030204c8bd62
I20260929 02:49:54.584556 12216 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a7e4401e9e44068ca9030204c8bd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:54 UTC) from {username='oryx', principal='oryx/127.4.197.188@KRBTEST.COM'} at 127.0.0.1:36464
I20260929 02:49:54.584554 11927 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a7e4401e9e44068ca9030204c8bd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:54 UTC) from {username='oryx', principal='oryx/127.4.197.188@KRBTEST.COM'} at 127.0.0.1:58256
I20260929 02:49:54.584719 12075 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a7e4401e9e44068ca9030204c8bd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:54 UTC) from {username='oryx', principal='oryx/127.4.197.188@KRBTEST.COM'} at 127.0.0.1:47526
I20260929 02:49:54.586239 12215 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a7e4401e9e44068ca9030204c8bd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:54 UTC) from {username='oryx', principal='oryx/127.4.197.188@KRBTEST.COM'} at 127.0.0.1:36464
I20260929 02:49:54.588009 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 11785
I20260929 02:49:54.589898 12396 tablet_replica.cc:330] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e: stopping tablet replica
I20260929 02:49:54.590029 12397 tablet_replica.cc:330] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7: stopping tablet replica
I20260929 02:49:54.590090 12396 raft_consensus.cc:2229] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:54.590234 12397 raft_consensus.cc:2229] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:54.590510 12396 raft_consensus.cc:2258] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:54.590521 12397 raft_consensus.cc:2258] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:54.591476 12396 ts_tablet_manager.cc:1905] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:54.591634 12397 ts_tablet_manager.cc:1905] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:54.596405 12398 tablet_replica.cc:330] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1: stopping tablet replica
I20260929 02:49:54.596632 12398 raft_consensus.cc:2229] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:54.596962 12398 raft_consensus.cc:2258] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:54.597961 12396 ts_tablet_manager.cc:1918] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:54.598090 12397 ts_tablet_manager.cc:1918] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:54.598170 12396 log.cc:1196] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/32a7e4401e9e44068ca9030204c8bd62
I20260929 02:49:54.598182 12398 ts_tablet_manager.cc:1905] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:54.598747 12396 ts_tablet_manager.cc:1939] T 32a7e4401e9e44068ca9030204c8bd62 P bbc41f9a5e014994be48f46fab7d976e: Deleting consensus metadata
I20260929 02:49:54.598304 12397 log.cc:1196] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/32a7e4401e9e44068ca9030204c8bd62
I20260929 02:49:54.599872 12397 ts_tablet_manager.cc:1939] T 32a7e4401e9e44068ca9030204c8bd62 P af8df5fbcb6e46e09832e71a7ef19da7: Deleting consensus metadata
I20260929 02:49:54.600476 11800 catalog_manager.cc:4841] TS bbc41f9a5e014994be48f46fab7d976e (127.4.197.131:40099): tablet 32a7e4401e9e44068ca9030204c8bd62 (table test-table [id=ade9a790128644d2902fa992c3477dad]) successfully deleted
W20260929 02:49:54.600903 11800 catalog_manager.cc:4808] TS bbc41f9a5e014994be48f46fab7d976e (127.4.197.131:40099): delete failed for tablet 32a7e4401e9e44068ca9030204c8bd62 because the tablet was not found. No further retry: Not found: Tablet not found: 32a7e4401e9e44068ca9030204c8bd62
I20260929 02:49:54.601140 11800 catalog_manager.cc:4841] TS af8df5fbcb6e46e09832e71a7ef19da7 (127.4.197.129:33895): tablet 32a7e4401e9e44068ca9030204c8bd62 (table test-table [id=ade9a790128644d2902fa992c3477dad]) successfully deleted
I20260929 02:49:54.603742 12398 ts_tablet_manager.cc:1918] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:54.603924 12398 log.cc:1196] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/32a7e4401e9e44068ca9030204c8bd62
I20260929 02:49:54.604322 12398 ts_tablet_manager.cc:1939] T 32a7e4401e9e44068ca9030204c8bd62 P 770e83e4819e49fabebce41221489ed1: Deleting consensus metadata
I20260929 02:49:54.605400 11798 catalog_manager.cc:4841] TS 770e83e4819e49fabebce41221489ed1 (127.4.197.130:39001): tablet 32a7e4401e9e44068ca9030204c8bd62 (table test-table [id=ade9a790128644d2902fa992c3477dad]) successfully deleted
I20260929 02:49:54.612577 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.589451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.4.197.129:33895)
0929 02:49:54.589683 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:54.589710 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:54.595820 (+ 6110us) server_negotiation.cc:407] Connection header received
0929 02:49:54.596004 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:54.596009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:54.596106 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:54.596222 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:54.598129 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.599210 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.603030 (+ 3820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.603664 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.603816 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.607048 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:54.607063 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:54.607078 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:54.607114 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:54.609904 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:54.610707 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:54.610714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:54.610717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:54.610879 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:54.611240 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:54.611243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:54.611245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:54.611635 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:54.611864 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:54.612142 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 02:49:54.612313 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:49:54.613235 11987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:54.704845 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 11785
I20260929 02:49:54.718861 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 12003
I20260929 02:49:54.731457 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.719588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.4.197.130:39001)
0929 02:49:54.719790 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:54.719794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:54.719812 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:49:54.719951 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:54.719955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:54.720017 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:54.720139 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:54.721295 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.721965 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.723149 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.723558 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.723648 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.725942 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:54.725958 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:54.725973 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:54.726009 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:54.728751 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:54.729378 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:54.729386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:54.729388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:54.729532 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:54.729919 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:54.729923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:54.729925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:54.730274 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:54.730510 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:54.730812 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:49:54.731132 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:49:54.732074 12135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:54.821506 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 12003
I20260929 02:49:54.835287 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 12153
I20260929 02:49:54.848904 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.836148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.4.197.131:40099)
0929 02:49:54.836341 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:54.836347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:54.836412 (+ 65us) server_negotiation.cc:407] Connection header received
0929 02:49:54.836612 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:54.836640 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:54.836707 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:54.836795 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:54.838137 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.838889 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.840312 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:54.840745 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:54.840856 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:54.843489 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:54.843504 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:54.843520 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:54.843557 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:54.846296 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:54.846941 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:54.846952 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:54.846956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:54.847105 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:54.847521 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:54.847524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:54.847526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:54.847898 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:54.848113 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:54.848447 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 02:49:54.848609 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:49:54.849561 12276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:54.898339 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.897918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40768)
0929 02:49:54.898106 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.898199 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 02:49:54.899299 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:54.898985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:38301 (local address 127.0.0.1:40784)
0929 02:49:54.899130 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:54.899200 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:38301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 02:49:54.967587 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 12153
I20260929 02:49:54.984566 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 11487
I20260929 02:49:54.985599 11654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:55.101650 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 11487
I20260929 02:49:55.117188 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 11588
I20260929 02:49:55.118070 11843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:55.236570 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 11588
2026-09-29T02:49:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8893 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:55 dist-test-slave-t6c1 krb5kdc[12407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:55 dist-test-slave-t6c1 krb5kdc[12407](info): set up 2 sockets
Sep 29 02:49:55 dist-test-slave-t6c1 krb5kdc[12407](info): commencing operation
krb5kdc: starting...
W20260929 02:49:56.247839 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:49:56 dist-test-slave-t6c1 krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.190.keytab.
Entry for principal oryx/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.190.keytab.
I20260929 02:49:56.453904 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45593
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.190.keytab
--principal=oryx/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:45593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:56.764957 12420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:56.765251 12420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:56.765484 12420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:56.772751 12420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:56.772859 12420 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:49:56.772923 12420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:56.773013 12420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:56.773073 12420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:56.773154 12420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:56.783202 12420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:45593
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.190.keytab
--principal=oryx/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:56.784499 12420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:56.785828 12420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:56.796872 12426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:56.796931 12425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:56.796947 12428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:56.797545 12420 server_base.cc:1034] running on GCE node
I20260929 02:49:56.798157 12420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:56.799433 12420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:56.800633 12420 hybrid_clock.cc:648] HybridClock initialized: now 1790650196800581 us; error 26 us; skew 500 ppm
I20260929 02:49:56.800902 12420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:56 dist-test-slave-t6c1 krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650196, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:56.803673 12420 init.cc:377] Logged in from keytab as oryx/127.4.197.190@KRBTEST.COM (short username oryx)
I20260929 02:49:56.804948 12420 webserver.cc:460] Webserver started at http://127.4.197.190:37821/ using document root <none> and password file <none>
I20260929 02:49:56.805558 12420 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:56.805641 12420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:56.805963 12420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:56.808681 12420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "5d13112991eb4af68bd0676cfe326359"
format_stamp: "Formatted at 2026-09-29 02:49:56 on dist-test-slave-t6c1"
server_key: "c080fb7e54d8d5555392e882d4265a51"
server_key_iv: "0278b658624f5bb15b7d27a734ed52ce"
server_key_version: "encryptionkey@0"
I20260929 02:49:56.809386 12420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "5d13112991eb4af68bd0676cfe326359"
format_stamp: "Formatted at 2026-09-29 02:49:56 on dist-test-slave-t6c1"
server_key: "c080fb7e54d8d5555392e882d4265a51"
server_key_iv: "0278b658624f5bb15b7d27a734ed52ce"
server_key_version: "encryptionkey@0"
I20260929 02:49:56.813974 12420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:49:56.817039 12435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:56.818205 12420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 02:49:56.818344 12420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "5d13112991eb4af68bd0676cfe326359"
format_stamp: "Formatted at 2026-09-29 02:49:56 on dist-test-slave-t6c1"
server_key: "c080fb7e54d8d5555392e882d4265a51"
server_key_iv: "0278b658624f5bb15b7d27a734ed52ce"
server_key_version: "encryptionkey@0"
I20260929 02:49:56.818516 12420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:56.849447 12420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:56.853672 12420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:56.853924 12420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:56.873584 12420 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:45593
I20260929 02:49:56.873607 12496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:45593 every 8 connection(s)
I20260929 02:49:56.874998 12420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:56.878775 12497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:56.882161 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 12420
I20260929 02:49:56.882382 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:49:56.883210 4886 external_mini_cluster.cc:1428] Setting key eaaad1547ef2ff7f79b8c2a8fe0c707b
I20260929 02:49:56.891317 12497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359: Bootstrap starting.
I20260929 02:49:56.894433 12497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:56.895475 12497 log.cc:824] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:49:56 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:56.898414 12497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359: No bootstrap required, opened a new log
I20260929 02:49:56.904433 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:56.885364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.4.197.190:45593)
0929 02:49:56.886115 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:56.886129 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:56.886171 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:49:56.887889 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:56.887913 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:56.888362 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:56.888790 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:56.890170 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:56.891530 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:56.892747 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:56.893428 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:56.893602 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:56.897691 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:56.897699 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:56.897713 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:56.897753 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:56.900630 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:56.901362 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:56.901368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:56.901374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:56.901533 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:56.901889 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:56.901892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:56.901894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:56.902400 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:56.902602 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:56.903063 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 02:49:56.903401 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":326,"threads_started":1}
I20260929 02:49:56.906699 12497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13112991eb4af68bd0676cfe326359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45593 } }
I20260929 02:49:56.907053 12497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:56.907135 12497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d13112991eb4af68bd0676cfe326359, State: Initialized, Role: FOLLOWER
I20260929 02:49:56.907676 12497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13112991eb4af68bd0676cfe326359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45593 } }
I20260929 02:49:56.907892 12497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:56.908013 12497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:56.908164 12497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:56.910187 12497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13112991eb4af68bd0676cfe326359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45593 } }
I20260929 02:49:56.910641 12497 leader_election.cc:302] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d13112991eb4af68bd0676cfe326359; no voters:
I20260929 02:49:56.911252 12497 leader_election.cc:288] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:56.911355 12502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:56.911963 12502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [term 1 LEADER]: Becoming Leader. State: Replica: 5d13112991eb4af68bd0676cfe326359, State: Running, Role: LEADER
I20260929 02:49:56.912492 12502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13112991eb4af68bd0676cfe326359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45593 } }
I20260929 02:49:56.913268 12497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:56.915009 12504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d13112991eb4af68bd0676cfe326359. Latest consensus state: current_term: 1 leader_uuid: "5d13112991eb4af68bd0676cfe326359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13112991eb4af68bd0676cfe326359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45593 } } }
I20260929 02:49:56.915359 12504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:56.920090 12503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d13112991eb4af68bd0676cfe326359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13112991eb4af68bd0676cfe326359" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45593 } } }
I20260929 02:49:56.920290 12503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359 [sys.catalog]: This master's current role is: LEADER
W20260929 02:49:56.924645 12518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:49:56.924751 12518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:49:56.924862 12507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:56.926376 12507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:56.933784 12507 catalog_manager.cc:1347] Generated new cluster ID: c35657141cee46be90248feefb4cfc9b
I20260929 02:49:56.933902 12507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:56.968657 12507 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:56.969926 12507 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:56.979061 12507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359: Generated new TSK 0
I20260929 02:49:56.979985 12507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab.
Entry for principal oryx/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab.
I20260929 02:49:57.032730 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45593
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab
--principal=oryx/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:57.352216 12523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:57.352546 12523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:57.352699 12523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:57.360168 12523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:57.360317 12523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:57.360446 12523 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:49:57.360555 12523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:49:57.370429 12523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab
--principal=oryx/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.197.190:45593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:57.371785 12523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:57.373085 12523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:57.387174 12528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:57.387610 12529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:57.388482 12531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:57.389345 12523 server_base.cc:1034] running on GCE node
I20260929 02:49:57.389945 12523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:57.390869 12523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:57.392108 12523 hybrid_clock.cc:648] HybridClock initialized: now 1790650197392034 us; error 52 us; skew 500 ppm
I20260929 02:49:57.392406 12523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:57 dist-test-slave-t6c1 krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650197, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:57.395378 12523 init.cc:377] Logged in from keytab as oryx/127.4.197.129@KRBTEST.COM (short username oryx)
I20260929 02:49:57.396709 12523 webserver.cc:460] Webserver started at http://127.4.197.129:37591/ using document root <none> and password file <none>
I20260929 02:49:57.397380 12523 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:57.397476 12523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:57.397760 12523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:57.400596 12523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "f5ba865194ad474081b82a7164092909"
format_stamp: "Formatted at 2026-09-29 02:49:57 on dist-test-slave-t6c1"
server_key: "7f6ff21dfa939d2e99399a39e7399fa4"
server_key_iv: "6b11d00475206cd20a1fa4356a540597"
server_key_version: "encryptionkey@0"
I20260929 02:49:57.401330 12523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "f5ba865194ad474081b82a7164092909"
format_stamp: "Formatted at 2026-09-29 02:49:57 on dist-test-slave-t6c1"
server_key: "7f6ff21dfa939d2e99399a39e7399fa4"
server_key_iv: "6b11d00475206cd20a1fa4356a540597"
server_key_version: "encryptionkey@0"
I20260929 02:49:57.406093 12523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:49:57.409224 12538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:57.410421 12523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:57.410552 12523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "f5ba865194ad474081b82a7164092909"
format_stamp: "Formatted at 2026-09-29 02:49:57 on dist-test-slave-t6c1"
server_key: "7f6ff21dfa939d2e99399a39e7399fa4"
server_key_iv: "6b11d00475206cd20a1fa4356a540597"
server_key_version: "encryptionkey@0"
I20260929 02:49:57.410732 12523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:57.427228 12523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:57.431229 12523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:57.431484 12523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:57.434068 12523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:57.434151 12523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:57.434281 12523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:57.434393 12523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 02:49:57 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650197, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:57.455958 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:57.434568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:49951 (local address 127.4.197.190:45593)
0929 02:49:57.435299 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:57.435306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:57.436318 (+ 1012us) server_negotiation.cc:407] Connection header received
0929 02:49:57.438471 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:57.438477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:57.438572 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:57.438688 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:57.440965 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.442135 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.444402 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.444976 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.445093 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:57.449276 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:57.449281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:57.449284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:57.449320 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:57.452667 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:57.453450 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:57.453457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:57.453461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:57.453629 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:57.454031 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:57.454036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:57.454040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:57.454412 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:57.454564 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:57.455345 (+ 781us) server_negotiation.cc:299] Negotiation successful
0929 02:49:57.455577 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":494,"threads_started":1}
I20260929 02:49:57.457085 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:57.435299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45593 (local address 127.4.197.129:49951)
0929 02:49:57.436138 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:57.436185 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:57.438093 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:57.439070 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:57.439081 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:57.439901 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:57.440741 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:57.440778 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.442315 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.442324 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:57.444230 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:57.444246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.445208 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.445223 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:57.445445 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:57.446195 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:57.446229 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:57.449040 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:57.452829 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:57.452839 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:57.452858 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:57.453325 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:57.453731 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:57.453735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:57.453737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:57.453932 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:57.454528 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:57.454534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:57.454914 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 02:49:57.455246 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 02:49:57.455810 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":347,"threads_started":1}
I20260929 02:49:57.478881 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:57.464866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:33569 (local address 127.4.197.190:45593)
0929 02:49:57.465040 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:57.465045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:57.465844 (+ 799us) server_negotiation.cc:407] Connection header received
0929 02:49:57.466009 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:57.466014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:57.466079 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:57.466158 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:57.467193 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.468186 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.470068 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.470694 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.470847 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:57.472632 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:57.472639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:57.472645 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:57.472681 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:57.476007 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:57.476874 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:57.476881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:57.476885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:57.477063 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:57.477384 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:57.477388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:57.477392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:57.477748 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:57.477920 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:57.478478 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 02:49:57.478650 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:49:57.479355 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:57.464962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45593 (local address 127.4.197.129:33569)
0929 02:49:57.465730 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:57.465755 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:57.465894 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:57.466195 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:57.466200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:57.466710 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:57.467054 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:57.467069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.468357 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.468362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:57.469919 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:57.469933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.470913 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.470942 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:57.471108 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:57.471700 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:57.471720 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:57.472444 (+ 724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:57.476190 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:57.476195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:57.476199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:57.476756 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:57.477163 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:57.477167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:57.477169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:57.477300 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:57.477991 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:57.477998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:57.478216 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 02:49:57.478384 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 02:49:57.478821 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":410,"threads_started":1}
I20260929 02:49:57.484447 12523 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:36323
I20260929 02:49:57.484467 12658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:36323 every 8 connection(s)
I20260929 02:49:57.485659 12523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:57.487192 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 12523
I20260929 02:49:57.487371 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:57.487989 4886 external_mini_cluster.cc:1428] Setting key 5545d837d0b9b704b313b013cd13b58e
I20260929 02:49:57.489398 12659 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45593
I20260929 02:49:57.489652 12659 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:57.490327 12659 heartbeater.cc:511] Master 127.4.197.190:45593 requested a full tablet report, sending...
I20260929 02:49:57.492126 12452 ts_manager.cc:194] Registered new tserver with Master: f5ba865194ad474081b82a7164092909 (127.4.197.129:36323)
I20260929 02:49:57.493510 12452 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:49951
WARNING: no policy specified for oryx/127.4.197.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab.
Entry for principal oryx/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab.
I20260929 02:49:57.529784 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45593
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab
--principal=oryx/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:57.848343 12663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:57.848628 12663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:57.848734 12663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:57.855958 12663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:57.856081 12663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:57.856166 12663 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:49:57.856246 12663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:49:57.866235 12663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab
--principal=oryx/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.197.190:45593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:57.867609 12663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:57.868911 12663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:57.881762 12668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:57.881798 12671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:57.881774 12669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:57.882841 12663 server_base.cc:1034] running on GCE node
I20260929 02:49:57.883451 12663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:57.884406 12663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:57.885710 12663 hybrid_clock.cc:648] HybridClock initialized: now 1790650197885599 us; error 55 us; skew 500 ppm
I20260929 02:49:57.886014 12663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:57 dist-test-slave-t6c1 krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650197, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:57.889264 12663 init.cc:377] Logged in from keytab as oryx/127.4.197.130@KRBTEST.COM (short username oryx)
I20260929 02:49:57.890697 12663 webserver.cc:460] Webserver started at http://127.4.197.130:43507/ using document root <none> and password file <none>
I20260929 02:49:57.891389 12663 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:57.891487 12663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:57.891783 12663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:57.894506 12663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "456ab55e11a24deda4b6020b92d4afc0"
format_stamp: "Formatted at 2026-09-29 02:49:57 on dist-test-slave-t6c1"
server_key: "f85be2d2362e59b7273ac50b70f1df2d"
server_key_iv: "103cb6ce457596eaec98f03a08c26074"
server_key_version: "encryptionkey@0"
I20260929 02:49:57.895277 12663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "456ab55e11a24deda4b6020b92d4afc0"
format_stamp: "Formatted at 2026-09-29 02:49:57 on dist-test-slave-t6c1"
server_key: "f85be2d2362e59b7273ac50b70f1df2d"
server_key_iv: "103cb6ce457596eaec98f03a08c26074"
server_key_version: "encryptionkey@0"
I20260929 02:49:57.899884 12663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:49:57.903084 12678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:57.904332 12663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:49:57.904467 12663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "456ab55e11a24deda4b6020b92d4afc0"
format_stamp: "Formatted at 2026-09-29 02:49:57 on dist-test-slave-t6c1"
server_key: "f85be2d2362e59b7273ac50b70f1df2d"
server_key_iv: "103cb6ce457596eaec98f03a08c26074"
server_key_version: "encryptionkey@0"
I20260929 02:49:57.904641 12663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:57.932175 12663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:57.936427 12663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:57.936693 12663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:57.939447 12663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:57.939553 12663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:57.939678 12663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:57.939744 12663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:49:57 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650197, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:57.962286 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:57.940137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:52391 (local address 127.4.197.190:45593)
0929 02:49:57.940308 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:57.940315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:57.942040 (+ 1725us) server_negotiation.cc:407] Connection header received
0929 02:49:57.944274 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:57.944280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:57.944366 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:57.944452 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:57.946630 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.947889 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.950167 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.950668 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.950796 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:57.955564 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:57.955570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:57.955573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:57.955607 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:57.958688 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:57.959661 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:57.959668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:57.959673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:57.959830 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:57.960419 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:57.960425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:57.960429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:57.960725 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:57.960880 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:57.961719 (+ 839us) server_negotiation.cc:299] Negotiation successful
0929 02:49:57.961943 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 02:49:57.963557 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:57.940923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45593 (local address 127.4.197.130:52391)
0929 02:49:57.941850 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:57.941892 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:57.943934 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:57.944735 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:57.944743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:57.945523 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:57.946393 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:57.946428 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.948099 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.948105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:57.949999 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:57.950012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.951109 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.951121 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:57.951337 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:57.952097 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:57.952145 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:57.955316 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:57.958865 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:57.958875 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:57.958897 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:57.959525 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:57.959985 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:57.959990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:57.959998 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:57.960299 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:57.960857 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:57.960865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:57.961348 (+ 483us) client_negotiation.cc:769] Sending connection context
0929 02:49:57.961619 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 02:49:57.962176 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":403,"threads_started":1}
I20260929 02:49:57.986321 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:57.973414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:52441 (local address 127.4.197.190:45593)
0929 02:49:57.973581 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:57.973585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:57.974210 (+ 625us) server_negotiation.cc:407] Connection header received
0929 02:49:57.974372 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:57.974377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:57.974442 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:57.974520 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:57.975694 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.976689 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.978219 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.978640 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.978759 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:57.981052 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:57.981056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:57.981060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:57.981092 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:57.983677 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:57.984491 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:57.984496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:57.984499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:57.984676 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:57.985056 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:57.985060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:57.985064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:57.985365 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:57.985471 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:57.985931 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 02:49:57.986097 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:49:57.986704 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:57.973562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45593 (local address 127.4.197.130:52441)
0929 02:49:57.974089 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:57.974107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:57.974257 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:57.974557 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:57.974561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:57.975071 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:57.975536 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:57.975554 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.976821 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.976826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:57.978080 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:57.978091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:57.978841 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:57.978849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:57.979076 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:57.979748 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:57.979772 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:57.980866 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:57.983798 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:57.983804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:57.983809 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:57.984365 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:57.984781 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:57.984786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:57.984790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:57.984951 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:57.985468 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:57.985474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:57.985693 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 02:49:57.985847 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 02:49:57.986204 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":281,"threads_started":1}
I20260929 02:49:57.992758 12663 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:41013
I20260929 02:49:57.992775 12797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:41013 every 8 connection(s)
I20260929 02:49:57.993865 12663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:57.995703 12798 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45593
I20260929 02:49:57.995909 12798 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:57.996477 12798 heartbeater.cc:511] Master 127.4.197.190:45593 requested a full tablet report, sending...
I20260929 02:49:57.997629 12452 ts_manager.cc:194] Registered new tserver with Master: 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013)
I20260929 02:49:57.998457 12452 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:52391
I20260929 02:49:57.999300 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 12663
I20260929 02:49:57.999449 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:57.999846 4886 external_mini_cluster.cc:1428] Setting key d271c8f81c04739d0d10ef215adbf507
WARNING: no policy specified for oryx/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab.
Entry for principal oryx/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab.
I20260929 02:49:58.040220 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45593
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab
--principal=oryx/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:58.352808 12802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:58.353098 12802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:58.353195 12802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:58.360426 12802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:58.360532 12802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:58.360622 12802 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:49:58.360702 12802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:49:58.370658 12802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab
--principal=oryx/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.197.190:45593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:49:58.372045 12802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:58.373396 12802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:58.386633 12807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:58.386703 12810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:58.386741 12808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:58.386873 12802 server_base.cc:1034] running on GCE node
I20260929 02:49:58.387866 12802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:58.388739 12802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:58.389961 12802 hybrid_clock.cc:648] HybridClock initialized: now 1790650198389883 us; error 52 us; skew 500 ppm
I20260929 02:49:58.390264 12802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:58 dist-test-slave-t6c1 krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650198, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:58.393340 12802 init.cc:377] Logged in from keytab as oryx/127.4.197.131@KRBTEST.COM (short username oryx)
I20260929 02:49:58.394691 12802 webserver.cc:460] Webserver started at http://127.4.197.131:36541/ using document root <none> and password file <none>
I20260929 02:49:58.395387 12802 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:58.395478 12802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:58.395756 12802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:58.398680 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "29180caa0a1840dd856b88202e247789"
format_stamp: "Formatted at 2026-09-29 02:49:58 on dist-test-slave-t6c1"
server_key: "4574827d538ac753dcc812f913d01738"
server_key_iv: "75829712e2a60ae70399993c59ff1875"
server_key_version: "encryptionkey@0"
I20260929 02:49:58.399559 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "29180caa0a1840dd856b88202e247789"
format_stamp: "Formatted at 2026-09-29 02:49:58 on dist-test-slave-t6c1"
server_key: "4574827d538ac753dcc812f913d01738"
server_key_iv: "75829712e2a60ae70399993c59ff1875"
server_key_version: "encryptionkey@0"
I20260929 02:49:58.404650 12802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:49:58.408207 12817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:58.409735 12802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:58.409904 12802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "29180caa0a1840dd856b88202e247789"
format_stamp: "Formatted at 2026-09-29 02:49:58 on dist-test-slave-t6c1"
server_key: "4574827d538ac753dcc812f913d01738"
server_key_iv: "75829712e2a60ae70399993c59ff1875"
server_key_version: "encryptionkey@0"
I20260929 02:49:58.410097 12802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:58.433117 12802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:58.436820 12802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:58.437055 12802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:58.439708 12802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:58.439790 12802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:58.439951 12802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:58.440039 12802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:49:58 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650198, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:58.461791 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:58.440063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:49029 (local address 127.4.197.190:45593)
0929 02:49:58.440252 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:58.440258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:58.441678 (+ 1420us) server_negotiation.cc:407] Connection header received
0929 02:49:58.444286 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:58.444293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:58.444385 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:58.444462 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:58.446818 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:58.447791 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:58.449862 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:58.450386 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:58.450511 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:58.455055 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:58.455061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:58.455065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:58.455104 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:58.458119 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:58.458966 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:58.458971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:58.458975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:58.459154 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:58.459706 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:58.459711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:58.459715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:58.460054 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:58.460188 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:58.461194 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0929 02:49:58.461444 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:49:58.463358 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:58.440660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45593 (local address 127.4.197.131:49029)
0929 02:49:58.441497 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:58.441550 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:58.443957 (+ 2407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:58.444980 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:58.444991 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:58.445809 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:58.446627 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:58.446665 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:58.447946 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:58.447952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:58.449712 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:58.449727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:58.450782 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:58.450787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:58.451088 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:58.451803 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:58.451848 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:58.454747 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:58.458243 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:58.458251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:58.458265 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:58.458819 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:58.459282 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:58.459287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:58.459291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:58.459548 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:58.460281 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:58.460289 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:58.460747 (+ 458us) client_negotiation.cc:769] Sending connection context
0929 02:49:58.461098 (+ 351us) client_negotiation.cc:240] Negotiation successful
0929 02:49:58.461656 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":343,"threads_started":1}
I20260929 02:49:58.479415 12802 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:36687
I20260929 02:49:58.479481 12936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:36687 every 8 connection(s)
I20260929 02:49:58.480633 12802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:58.482728 12937 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45593
I20260929 02:49:58.482964 12937 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:58.483783 12937 heartbeater.cc:511] Master 127.4.197.190:45593 requested a full tablet report, sending...
I20260929 02:49:58.484218 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:58.470119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:51047 (local address 127.4.197.190:45593)
0929 02:49:58.470319 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:58.470326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:58.470978 (+ 652us) server_negotiation.cc:407] Connection header received
0929 02:49:58.471207 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:58.471210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:58.471290 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:58.471381 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:58.472615 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:58.473347 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:58.475468 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:58.475971 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:58.476096 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:58.478341 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:58.478346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:58.478350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:58.478382 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:58.481120 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:58.482141 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:58.482146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:58.482150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:58.482294 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:58.482687 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:58.482693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:58.482697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:58.483071 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:58.483187 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:58.483672 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 02:49:58.483893 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:49:58.484356 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:58.470074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45593 (local address 127.4.197.131:51047)
0929 02:49:58.470819 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:58.470847 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:58.471080 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:58.471437 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:58.471442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:58.472074 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:58.472476 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:58.472490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:58.473493 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:58.473498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:58.475302 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:58.475323 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:58.476206 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:58.476213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:58.476450 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:58.477264 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:58.477292 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:58.478146 (+ 854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:58.481248 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:58.481253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:58.481258 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:58.481843 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:58.482441 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:58.482445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:58.482449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:58.482585 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:58.483181 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:58.483187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:58.483404 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 02:49:58.483559 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 02:49:58.483912 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":380,"threads_started":1}
I20260929 02:49:58.485544 12451 ts_manager.cc:194] Registered new tserver with Master: 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687)
I20260929 02:49:58.486689 12451 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:49029
I20260929 02:49:58.489462 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 12802
I20260929 02:49:58.489600 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:58.490013 4886 external_mini_cluster.cc:1428] Setting key 6f5ea85779a0ed79f6e238d339fa3d12
I20260929 02:49:58.492695 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:58.496116 12659 heartbeater.cc:503] Master 127.4.197.190:45593 was elected leader, sending a full tablet report...
Sep 29 02:49:58 dist-test-slave-t6c1 krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:49:58 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:49:58 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:49:59.000926 12798 heartbeater.cc:503] Master 127.4.197.190:45593 was elected leader, sending a full tablet report...
I20260929 02:49:59.489800 12937 heartbeater.cc:503] Master 127.4.197.190:45593 was elected leader, sending a full tablet report...
W20260929 02:49:59.502128 12945 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.967s user 0.000s sys 0.012s
W20260929 02:49:59.503178 12547 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:49:58.529461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42272 (local address 127.4.197.190:45593)
0929 02:49:58.529697 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:58.529704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:58.529861 (+ 157us) server_negotiation.cc:407] Connection header received
0929 02:49:58.530051 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:58.530055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:58.530119 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:58.530201 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:58.531630 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:58.532372 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:58.533625 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:58.534069 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:58.534180 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.502862 (+968682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42272: BlockingRecv error: recv got EOF from 127.0.0.1:42272 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 02:49:59 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:59.519342 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.506715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.4.197.190:45593)
0929 02:49:59.506885 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:59.506890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:59.507202 (+ 312us) server_negotiation.cc:407] Connection header received
0929 02:49:59.507331 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:59.507335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:59.507421 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:59.507509 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:59.508723 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.509428 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.510546 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.511021 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.511113 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.514581 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:59.514585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:59.514589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:59.514649 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:59.516997 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:59.517593 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:59.517596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:59.517599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:59.517738 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:59.518051 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:59.518054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:59.518055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:59.518293 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:59.518585 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:59.518813 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 02:49:59.519054 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:49:59.523236 12451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:59.529794 12451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:59.565599 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.541140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36323 (local address 127.0.0.1:36144)
0929 02:49:59.557726 (+ 16586us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:59.557751 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:59.557893 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:59.558576 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:59.558584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:59.558611 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:59.559020 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.559030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.560603 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.560607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.562717 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.562731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.564597 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.564603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.564791 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.564849 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 02:49:59.564973 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 02:49:59.565161 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16465,"spinlock_wait_cycles":768,"thread_start_us":16310,"threads_started":1}
I20260929 02:49:59.565905 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.541403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.4.197.130:41013)
0929 02:49:59.558546 (+ 17143us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:59.558555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:59.558581 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:49:59.558666 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:59.558672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:59.558998 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:59.559232 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:59.559873 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.561328 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.563406 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.564827 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.564950 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.565192 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:59.565399 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 02:49:59.565547 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16992,"thread_start_us":16871,"threads_started":1}
I20260929 02:49:59.566140 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.540778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36028 (local address 127.4.197.131:36687)
0929 02:49:59.556019 (+ 15241us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:59.556030 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:59.557792 (+ 1762us) server_negotiation.cc:407] Connection header received
0929 02:49:59.558103 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:59.558111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:59.558397 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:59.558565 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:59.559626 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.561175 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.563613 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.564837 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.564967 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.565197 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:59.565719 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 02:49:59.565875 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15051,"thread_start_us":15264,"threads_started":1}
I20260929 02:49:59.566351 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.540869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:41013 (local address 127.0.0.1:40296)
0929 02:49:59.556840 (+ 15971us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:59.556886 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:59.557052 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:59.559212 (+ 2160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:59.559217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:59.559242 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:59.559639 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.559660 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.561462 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.561467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.563221 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.563235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.565007 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.565013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.565225 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.565257 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:49:59.565347 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 02:49:59.565472 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15712,"mutex_wait_us":18,"thread_start_us":15575,"threads_started":1}
I20260929 02:49:59.566834 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.540578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:36687 (local address 127.0.0.1:36028)
0929 02:49:59.557671 (+ 17093us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:59.557691 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:59.557905 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:59.558899 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:59.558904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:59.558947 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:59.559456 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.559470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.561322 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.561328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.563454 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.563466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.565335 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.565341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.565490 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.565520 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:49:59.566375 (+ 855us) client_negotiation.cc:240] Negotiation successful
0929 02:49:59.566509 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15947,"mutex_wait_us":26,"spinlock_wait_cycles":15232,"thread_start_us":15751,"threads_started":1}
I20260929 02:49:59.566900 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.541421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.4.197.129:36323)
0929 02:49:59.557189 (+ 15768us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:59.557254 (+ 65us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:59.558013 (+ 759us) server_negotiation.cc:407] Connection header received
0929 02:49:59.558129 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:59.558137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:59.558384 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:59.558574 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:59.559193 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.560480 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.562885 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.564398 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.564538 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.564813 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:59.565753 (+ 940us) server_negotiation.cc:299] Negotiation successful
0929 02:49:59.566638 (+ 885us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15566,"thread_start_us":15571,"threads_started":1}
I20260929 02:49:59.569836 12728 tablet_service.cc:1467] Processing CreateTablet for tablet 5c40f557abef49c3a2bfbc85c58363f0 (DEFAULT_TABLE table=test-table [id=ef1fe8c668a443a3b7691e695f936b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:59.570117 12589 tablet_service.cc:1467] Processing CreateTablet for tablet 5c40f557abef49c3a2bfbc85c58363f0 (DEFAULT_TABLE table=test-table [id=ef1fe8c668a443a3b7691e695f936b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:59.570426 12867 tablet_service.cc:1467] Processing CreateTablet for tablet 5c40f557abef49c3a2bfbc85c58363f0 (DEFAULT_TABLE table=test-table [id=ef1fe8c668a443a3b7691e695f936b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:59.571199 12728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c40f557abef49c3a2bfbc85c58363f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:59.571441 12589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c40f557abef49c3a2bfbc85c58363f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:59.571609 12867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c40f557abef49c3a2bfbc85c58363f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:59.581600 12962 tablet_bootstrap.cc:492] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: Bootstrap starting.
I20260929 02:49:59.584815 12962 tablet_bootstrap.cc:654] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:59.585237 12963 tablet_bootstrap.cc:492] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: Bootstrap starting.
I20260929 02:49:59.586032 12964 tablet_bootstrap.cc:492] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: Bootstrap starting.
I20260929 02:49:59.586262 12962 log.cc:824] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:59.589465 12963 tablet_bootstrap.cc:654] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:59.589524 12964 tablet_bootstrap.cc:654] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:59.589609 12962 tablet_bootstrap.cc:492] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: No bootstrap required, opened a new log
I20260929 02:49:59.589910 12962 ts_tablet_manager.cc:1397] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260929 02:49:59.590677 12963 log.cc:824] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:59.590806 12964 log.cc:824] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:59.595636 12963 tablet_bootstrap.cc:492] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: No bootstrap required, opened a new log
I20260929 02:49:59.595919 12963 ts_tablet_manager.cc:1397] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 02:49:59.597066 12964 tablet_bootstrap.cc:492] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: No bootstrap required, opened a new log
I20260929 02:49:59.597393 12964 ts_tablet_manager.cc:1397] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 02:49:59.598881 12962 raft_consensus.cc:348] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } }
I20260929 02:49:59.599326 12962 raft_consensus.cc:374] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:59.599436 12962 raft_consensus.cc:729] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456ab55e11a24deda4b6020b92d4afc0, State: Initialized, Role: FOLLOWER
I20260929 02:49:59.600026 12962 consensus_queue.cc:260] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } }
I20260929 02:49:59.601840 12962 ts_tablet_manager.cc:1428] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260929 02:49:59.606437 12963 raft_consensus.cc:348] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } }
I20260929 02:49:59.606964 12963 raft_consensus.cc:374] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:59.607076 12963 raft_consensus.cc:729] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ba865194ad474081b82a7164092909, State: Initialized, Role: FOLLOWER
I20260929 02:49:59.607795 12963 consensus_queue.cc:260] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } }
I20260929 02:49:59.607903 12964 raft_consensus.cc:348] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } }
I20260929 02:49:59.608390 12964 raft_consensus.cc:374] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:59.608539 12964 raft_consensus.cc:729] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29180caa0a1840dd856b88202e247789, State: Initialized, Role: FOLLOWER
I20260929 02:49:59.609167 12964 consensus_queue.cc:260] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } }
I20260929 02:49:59.609891 12963 ts_tablet_manager.cc:1428] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260929 02:49:59.610972 12964 ts_tablet_manager.cc:1428] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 02:49:59.689906 12970 raft_consensus.cc:482] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:59.690208 12970 raft_consensus.cc:504] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } }
I20260929 02:49:59.692039 12970 leader_election.cc:288] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013), f5ba865194ad474081b82a7164092909 (127.4.197.129:36323)
I20260929 02:49:59.700932 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.692558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:43615 (local address 127.4.197.129:36323)
0929 02:49:59.692781 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:59.692787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:59.693413 (+ 626us) server_negotiation.cc:407] Connection header received
0929 02:49:59.693635 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:59.693640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:59.693711 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:59.693779 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:59.694673 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.695832 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.698678 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.699895 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.700038 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.700110 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:59.700554 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 02:49:59.700685 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:49:59.700930 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.692571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:34077 (local address 127.4.197.130:41013)
0929 02:49:59.692773 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:59.692781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:59.693426 (+ 645us) server_negotiation.cc:407] Connection header received
0929 02:49:59.693639 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:59.693644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:59.693712 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:59.693770 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:59.694673 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.695832 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.698665 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.699878 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.700038 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.700110 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:59.700555 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 02:49:59.700682 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:49:59.701177 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.692336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:41013 (local address 127.4.197.131:34077)
0929 02:49:59.693286 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:59.693310 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:59.693473 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:59.693961 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:59.693966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:59.694056 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:59.694515 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.694525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.695964 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.695968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.698500 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.698515 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.700158 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.700163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.700343 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.700423 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 02:49:59.700519 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 02:49:59.700680 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":158,"spinlock_wait_cycles":25856,"thread_start_us":645,"threads_started":1}
I20260929 02:49:59.701197 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.692454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36323 (local address 127.4.197.131:43615)
0929 02:49:59.693321 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:59.693336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:59.693487 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:59.693961 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:59.693966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:59.693990 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:59.694489 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.694506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.695962 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.695968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.698507 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.698515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.700144 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.700151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.700350 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.700421 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 02:49:59.700519 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:49:59.700680 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":224,"spinlock_wait_cycles":16768,"thread_start_us":995,"threads_started":1}
I20260929 02:49:59.702414 12748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c40f557abef49c3a2bfbc85c58363f0" candidate_uuid: "29180caa0a1840dd856b88202e247789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456ab55e11a24deda4b6020b92d4afc0" is_pre_election: true
I20260929 02:49:59.702415 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c40f557abef49c3a2bfbc85c58363f0" candidate_uuid: "29180caa0a1840dd856b88202e247789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ba865194ad474081b82a7164092909" is_pre_election: true
I20260929 02:49:59.703001 12748 raft_consensus.cc:2454] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29180caa0a1840dd856b88202e247789 in term 0.
I20260929 02:49:59.703002 12609 raft_consensus.cc:2454] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29180caa0a1840dd856b88202e247789 in term 0.
I20260929 02:49:59.703706 12820 leader_election.cc:302] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29180caa0a1840dd856b88202e247789, f5ba865194ad474081b82a7164092909; no voters:
I20260929 02:49:59.704118 12970 raft_consensus.cc:2781] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:59.704268 12970 raft_consensus.cc:482] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:59.704360 12970 raft_consensus.cc:3037] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:59.706525 12970 raft_consensus.cc:504] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } }
I20260929 02:49:59.707063 12970 leader_election.cc:288] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [CANDIDATE]: Term 1 election: Requested vote from peers 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013), f5ba865194ad474081b82a7164092909 (127.4.197.129:36323)
I20260929 02:49:59.707415 12748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c40f557abef49c3a2bfbc85c58363f0" candidate_uuid: "29180caa0a1840dd856b88202e247789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456ab55e11a24deda4b6020b92d4afc0"
I20260929 02:49:59.707450 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c40f557abef49c3a2bfbc85c58363f0" candidate_uuid: "29180caa0a1840dd856b88202e247789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ba865194ad474081b82a7164092909"
I20260929 02:49:59.707578 12748 raft_consensus.cc:3037] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:59.707638 12609 raft_consensus.cc:3037] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:59.709619 12748 raft_consensus.cc:2454] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29180caa0a1840dd856b88202e247789 in term 1.
I20260929 02:49:59.709715 12609 raft_consensus.cc:2454] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29180caa0a1840dd856b88202e247789 in term 1.
I20260929 02:49:59.710036 12821 leader_election.cc:302] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29180caa0a1840dd856b88202e247789, 456ab55e11a24deda4b6020b92d4afc0; no voters:
I20260929 02:49:59.710286 12970 raft_consensus.cc:2781] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:59.710688 12970 raft_consensus.cc:686] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 1 LEADER]: Becoming Leader. State: Replica: 29180caa0a1840dd856b88202e247789, State: Running, Role: LEADER
I20260929 02:49:59.711125 12970 consensus_queue.cc:237] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } }
I20260929 02:49:59.715816 12451 catalog_manager.cc:5495] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29180caa0a1840dd856b88202e247789 (127.4.197.131). New cstate: current_term: 1 leader_uuid: "29180caa0a1840dd856b88202e247789" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } health_report { overall_health: UNKNOWN } } }
W20260929 02:49:59.733057 12938 tablet.cc:2369] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:49:59.738828 12660 tablet.cc:2369] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:59.744053 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.737600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.4.197.131:36687)
0929 02:49:59.739498 (+ 1898us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:59.739503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:59.739518 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:49:59.739597 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:59.739600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:59.739686 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:59.739804 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:59.740175 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.740862 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.742104 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.742497 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.742601 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.743031 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:59.743514 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:59.743639 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:59.743735 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 02:49:59.743815 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895}
W20260929 02:49:59.746457 12799 tablet.cc:2369] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:59.751364 12748 raft_consensus.cc:1261] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Refusing update from remote peer 29180caa0a1840dd856b88202e247789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:59.751364 12609 raft_consensus.cc:1261] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Refusing update from remote peer 29180caa0a1840dd856b88202e247789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:59.752368 12973 consensus_queue.cc:1035] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:59.752717 12970 consensus_queue.cc:1035] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:59.761969 12976 mvcc.cc:205] Tried to move back new op lower bound from 7334503218172678144 to 7334503218020069376. Current Snapshot: MvccSnapshot[applied={T|T < 7334503218172678144}]
I20260929 02:49:59.768635 12978 mvcc.cc:205] Tried to move back new op lower bound from 7334503218172678144 to 7334503218020069376. Current Snapshot: MvccSnapshot[applied={T|T < 7334503218172678144}]
I20260929 02:49:59.785084 12451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:59.785318 12451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:59.788653 12451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359: Sending DeleteTablet for 3 replicas of tablet 5c40f557abef49c3a2bfbc85c58363f0
I20260929 02:49:59.789609 12728 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c40f557abef49c3a2bfbc85c58363f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:59 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:40296
I20260929 02:49:59.789954 12867 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c40f557abef49c3a2bfbc85c58363f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:59 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:36028
I20260929 02:49:59.790570 12589 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c40f557abef49c3a2bfbc85c58363f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:59 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:36144
I20260929 02:49:59.790723 12451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:59.791476 12451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:59.792989 12987 tablet_replica.cc:330] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: stopping tablet replica
I20260929 02:49:59.793449 12987 raft_consensus.cc:2229] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:59.794108 12987 raft_consensus.cc:2258] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:59.798120 12987 ts_tablet_manager.cc:1905] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:59.800154 12727 tablet_service.cc:1467] Processing CreateTablet for tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92 (DEFAULT_TABLE table=test-table [id=9cd4b924348c4fc9a44d60f83f457c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:59.800448 12867 tablet_service.cc:1467] Processing CreateTablet for tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92 (DEFAULT_TABLE table=test-table [id=9cd4b924348c4fc9a44d60f83f457c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:59.800724 12727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:59.800865 12867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:59.804642 12588 tablet_service.cc:1467] Processing CreateTablet for tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92 (DEFAULT_TABLE table=test-table [id=9cd4b924348c4fc9a44d60f83f457c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:59.805296 12588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:59.805408 12988 tablet_replica.cc:330] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: stopping tablet replica
I20260929 02:49:59.805864 12988 raft_consensus.cc:2229] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:59.806387 12988 raft_consensus.cc:2258] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:59.807189 12986 tablet_replica.cc:330] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: stopping tablet replica
I20260929 02:49:59.807736 12986 raft_consensus.cc:2229] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:59.808276 12986 raft_consensus.cc:2258] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:59.810071 12988 ts_tablet_manager.cc:1905] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:59.812427 12963 tablet_bootstrap.cc:492] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909: Bootstrap starting.
I20260929 02:49:59.812422 12962 tablet_bootstrap.cc:492] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0: Bootstrap starting.
I20260929 02:49:59.813987 12962 tablet_bootstrap.cc:654] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:59.813995 12963 tablet_bootstrap.cc:654] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:59.815047 12964 tablet_bootstrap.cc:492] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789: Bootstrap starting.
I20260929 02:49:59.817333 12987 ts_tablet_manager.cc:1918] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:59.817564 12987 log.cc:1196] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/5c40f557abef49c3a2bfbc85c58363f0
I20260929 02:49:59.817819 12964 tablet_bootstrap.cc:654] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:59.818202 12987 ts_tablet_manager.cc:1939] T 5c40f557abef49c3a2bfbc85c58363f0 P 29180caa0a1840dd856b88202e247789: Deleting consensus metadata
I20260929 02:49:59.819932 12436 catalog_manager.cc:4841] TS 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687): tablet 5c40f557abef49c3a2bfbc85c58363f0 (table test-table [id=ef1fe8c668a443a3b7691e695f936b5a]) successfully deleted
I20260929 02:49:59.821172 12988 ts_tablet_manager.cc:1918] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:59.822402 12963 tablet_bootstrap.cc:492] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909: No bootstrap required, opened a new log
I20260929 02:49:59.822438 12988 log.cc:1196] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/5c40f557abef49c3a2bfbc85c58363f0
I20260929 02:49:59.822564 12963 ts_tablet_manager.cc:1397] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20260929 02:49:59.823187 12988 ts_tablet_manager.cc:1939] T 5c40f557abef49c3a2bfbc85c58363f0 P f5ba865194ad474081b82a7164092909: Deleting consensus metadata
I20260929 02:49:59.823479 12963 raft_consensus.cc:348] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.823689 12963 raft_consensus.cc:374] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:59.823781 12963 raft_consensus.cc:729] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ba865194ad474081b82a7164092909, State: Initialized, Role: FOLLOWER
I20260929 02:49:59.824044 12963 consensus_queue.cc:260] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.824967 12438 catalog_manager.cc:4841] TS f5ba865194ad474081b82a7164092909 (127.4.197.129:36323): tablet 5c40f557abef49c3a2bfbc85c58363f0 (table test-table [id=ef1fe8c668a443a3b7691e695f936b5a]) successfully deleted
I20260929 02:49:59.825380 12963 ts_tablet_manager.cc:1428] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:49:59.825486 12962 tablet_bootstrap.cc:492] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0: No bootstrap required, opened a new log
I20260929 02:49:59.825621 12962 ts_tablet_manager.cc:1397] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260929 02:49:59.826785 12962 raft_consensus.cc:348] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.827064 12962 raft_consensus.cc:374] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:59.827171 12962 raft_consensus.cc:729] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456ab55e11a24deda4b6020b92d4afc0, State: Initialized, Role: FOLLOWER
I20260929 02:49:59.827415 12986 ts_tablet_manager.cc:1905] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:59.827502 12962 consensus_queue.cc:260] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.828048 12962 ts_tablet_manager.cc:1428] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:59.830605 12964 tablet_bootstrap.cc:492] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789: No bootstrap required, opened a new log
I20260929 02:49:59.830809 12964 ts_tablet_manager.cc:1397] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20260929 02:49:59.831736 12964 raft_consensus.cc:348] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.832031 12964 raft_consensus.cc:374] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:59.832175 12964 raft_consensus.cc:729] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29180caa0a1840dd856b88202e247789, State: Initialized, Role: FOLLOWER
I20260929 02:49:59.832513 12964 consensus_queue.cc:260] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.833050 12964 ts_tablet_manager.cc:1428] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:59.838106 12986 ts_tablet_manager.cc:1918] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:59.838285 12986 log.cc:1196] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/5c40f557abef49c3a2bfbc85c58363f0
I20260929 02:49:59.838783 12986 ts_tablet_manager.cc:1939] T 5c40f557abef49c3a2bfbc85c58363f0 P 456ab55e11a24deda4b6020b92d4afc0: Deleting consensus metadata
I20260929 02:49:59.840205 12439 catalog_manager.cc:4841] TS 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013): tablet 5c40f557abef49c3a2bfbc85c58363f0 (table test-table [id=ef1fe8c668a443a3b7691e695f936b5a]) successfully deleted
I20260929 02:49:59.864845 12969 raft_consensus.cc:482] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:59.865072 12969 raft_consensus.cc:504] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.866712 12969 leader_election.cc:288] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013), 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687)
I20260929 02:49:59.874838 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.867030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:36687 (local address 127.4.197.129:60123)
0929 02:49:59.868319 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:59.868335 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:59.868463 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:59.868804 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:59.868809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:59.868833 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:59.869147 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.869158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.870606 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.870611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.872759 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.872775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.874021 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.874027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.874202 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.874226 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:49:59.874371 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 02:49:59.874514 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"thread_start_us":353,"threads_started":1}
I20260929 02:49:59.874843 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.867212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:60123 (local address 127.4.197.131:36687)
0929 02:49:59.867368 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:59.867375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:59.868429 (+ 1054us) server_negotiation.cc:407] Connection header received
0929 02:49:59.868575 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:59.868580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:59.868675 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:59.868779 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:59.869299 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.870444 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.872953 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.873831 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.873991 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.874060 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:59.874401 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 02:49:59.874527 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 02:49:59.876085 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.867336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:32793 (local address 127.4.197.130:41013)
0929 02:49:59.867513 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:59.867518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:59.867754 (+ 236us) server_negotiation.cc:407] Connection header received
0929 02:49:59.867955 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:59.867961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:59.868040 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:59.868123 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:59.868842 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.869730 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.872078 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.872896 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.873065 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.873154 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:59.875673 (+ 2519us) server_negotiation.cc:299] Negotiation successful
0929 02:49:59.875824 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:49:59.876094 12887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db7b4b19aad4d9fb97be7b3b9bdbd92" candidate_uuid: "f5ba865194ad474081b82a7164092909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29180caa0a1840dd856b88202e247789" is_pre_election: true
I20260929 02:49:59.876561 12887 raft_consensus.cc:2454] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ba865194ad474081b82a7164092909 in term 0.
I20260929 02:49:59.876911 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.866994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:41013 (local address 127.4.197.129:32793)
0929 02:49:59.867655 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:59.867670 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:59.867807 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:59.868282 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:59.868287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:59.868317 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:59.868675 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.868685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.869869 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.869872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.871919 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.871933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.873109 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.873116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.875490 (+ 2374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.875522 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:49:59.876424 (+ 902us) client_negotiation.cc:240] Negotiation successful
0929 02:49:59.876561 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":1,"thread_start_us":999,"threads_started":1}
I20260929 02:49:59.877382 12539 leader_election.cc:302] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29180caa0a1840dd856b88202e247789, f5ba865194ad474081b82a7164092909; no voters:
I20260929 02:49:59.877472 12748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db7b4b19aad4d9fb97be7b3b9bdbd92" candidate_uuid: "f5ba865194ad474081b82a7164092909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456ab55e11a24deda4b6020b92d4afc0" is_pre_election: true
I20260929 02:49:59.877775 12969 raft_consensus.cc:2781] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:59.877720 12748 raft_consensus.cc:2454] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ba865194ad474081b82a7164092909 in term 0.
I20260929 02:49:59.877875 12969 raft_consensus.cc:482] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:59.877966 12969 raft_consensus.cc:3037] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:59.880302 12969 raft_consensus.cc:504] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.880873 12969 leader_election.cc:288] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [CANDIDATE]: Term 1 election: Requested vote from peers 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013), 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687)
I20260929 02:49:59.881372 12748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db7b4b19aad4d9fb97be7b3b9bdbd92" candidate_uuid: "f5ba865194ad474081b82a7164092909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456ab55e11a24deda4b6020b92d4afc0"
I20260929 02:49:59.881489 12887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db7b4b19aad4d9fb97be7b3b9bdbd92" candidate_uuid: "f5ba865194ad474081b82a7164092909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29180caa0a1840dd856b88202e247789"
I20260929 02:49:59.881575 12748 raft_consensus.cc:3037] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:59.881677 12887 raft_consensus.cc:3037] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:59.884518 12748 raft_consensus.cc:2454] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ba865194ad474081b82a7164092909 in term 1.
I20260929 02:49:59.884778 12887 raft_consensus.cc:2454] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ba865194ad474081b82a7164092909 in term 1.
I20260929 02:49:59.885181 12542 leader_election.cc:302] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29180caa0a1840dd856b88202e247789, 456ab55e11a24deda4b6020b92d4afc0, f5ba865194ad474081b82a7164092909; no voters:
I20260929 02:49:59.885566 12969 raft_consensus.cc:2781] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:59.891255 12969 raft_consensus.cc:686] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 1 LEADER]: Becoming Leader. State: Replica: f5ba865194ad474081b82a7164092909, State: Running, Role: LEADER
I20260929 02:49:59.891686 12969 consensus_queue.cc:237] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.895395 12452 catalog_manager.cc:5495] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ba865194ad474081b82a7164092909 (127.4.197.129). New cstate: current_term: 1 leader_uuid: "f5ba865194ad474081b82a7164092909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:49:59 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650196, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.190@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:49:59.948696 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.929765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45593 (local address 127.0.0.1:42300)
0929 02:49:59.930408 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:59.930429 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:59.930583 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:59.930903 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:59.930908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:59.931854 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:59.932190 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.932202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.933210 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.933216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.934761 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:59.934776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.935611 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.935618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:59.935804 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.940095 (+ 4291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:59.940121 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:59.942779 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:59.945829 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:59.945846 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:59.945855 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:59.946343 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:59.946773 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:59.946778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:59.946783 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:59.946967 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:59.947521 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:59.947530 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:59.947706 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 02:49:59.947863 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 02:49:59.948174 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":370,"threads_started":1}
I20260929 02:49:59.948719 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:59.929747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.4.197.190:45593)
0929 02:49:59.929985 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:59.929992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:59.930520 (+ 528us) server_negotiation.cc:407] Connection header received
0929 02:49:59.930717 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:59.930721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:59.930780 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:59.930849 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:59.932340 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.933091 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.934902 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:59.935438 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:59.935530 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:59.943052 (+ 7522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:59.943058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:59.943062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:59.943100 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:59.945668 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:59.946476 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:59.946482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:59.946486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:59.946658 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:59.947073 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:59.947079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:59.947082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:59.947373 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:59.947540 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:59.947985 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 02:49:59.948313 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":110}
I20260929 02:49:59.954257 12452 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:42300:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:49:59.955292 12452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:59.962248 12588 tablet_service.cc:1467] Processing CreateTablet for tablet 377a85a8f77b44a4a610071b426e84b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=febf126a9311480f94467dabbdeab9f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:49:59.962556 12867 tablet_service.cc:1467] Processing CreateTablet for tablet 377a85a8f77b44a4a610071b426e84b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=febf126a9311480f94467dabbdeab9f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:49:59.962551 12728 tablet_service.cc:1467] Processing CreateTablet for tablet 377a85a8f77b44a4a610071b426e84b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=febf126a9311480f94467dabbdeab9f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:49:59.962829 12588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377a85a8f77b44a4a610071b426e84b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:59.963099 12867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377a85a8f77b44a4a610071b426e84b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:59.963093 12728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377a85a8f77b44a4a610071b426e84b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:59.970194 12962 tablet_bootstrap.cc:492] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: Bootstrap starting.
I20260929 02:49:59.970486 12963 tablet_bootstrap.cc:492] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: Bootstrap starting.
I20260929 02:49:59.971585 12962 tablet_bootstrap.cc:654] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:59.972165 12963 tablet_bootstrap.cc:654] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:59.975591 12964 tablet_bootstrap.cc:492] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: Bootstrap starting.
I20260929 02:49:59.977120 12964 tablet_bootstrap.cc:654] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:59.980526 12964 tablet_bootstrap.cc:492] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: No bootstrap required, opened a new log
I20260929 02:49:59.980731 12964 ts_tablet_manager.cc:1397] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 02:49:59.981173 12963 tablet_bootstrap.cc:492] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: No bootstrap required, opened a new log
I20260929 02:49:59.981340 12963 ts_tablet_manager.cc:1397] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 02:49:59.981364 12964 raft_consensus.cc:348] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.981690 12964 raft_consensus.cc:374] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:59.981832 12964 raft_consensus.cc:729] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29180caa0a1840dd856b88202e247789, State: Initialized, Role: FOLLOWER
I20260929 02:49:59.982067 12963 raft_consensus.cc:348] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.982147 12964 consensus_queue.cc:260] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.982385 12963 raft_consensus.cc:374] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:59.982496 12963 raft_consensus.cc:729] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ba865194ad474081b82a7164092909, State: Initialized, Role: FOLLOWER
I20260929 02:49:59.982869 12964 ts_tablet_manager.cc:1428] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:49:59.982707 12983 tablet_replica.cc:439] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } }
I20260929 02:49:59.983110 12983 tablet_replica.cc:442] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:59.983410 12963 consensus_queue.cc:260] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.983991 12969 tablet_replica.cc:439] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } }
I20260929 02:49:59.984288 12969 tablet_replica.cc:442] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:49:59.984326 12963 ts_tablet_manager.cc:1428] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:49:59.985519 12962 tablet_bootstrap.cc:492] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: No bootstrap required, opened a new log
I20260929 02:49:59.985671 12962 ts_tablet_manager.cc:1397] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20260929 02:49:59.986334 12962 raft_consensus.cc:348] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.986552 12962 raft_consensus.cc:374] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:59.986616 12962 raft_consensus.cc:729] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456ab55e11a24deda4b6020b92d4afc0, State: Initialized, Role: FOLLOWER
I20260929 02:49:59.986841 12962 consensus_queue.cc:260] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:49:59.987805 12962 ts_tablet_manager.cc:1428] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:49:59.987676 12968 tablet_replica.cc:439] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } }
I20260929 02:49:59.988058 12968 tablet_replica.cc:442] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:50:00.158774 12968 raft_consensus.cc:482] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:50:00.159122 12968 raft_consensus.cc:504] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.161214 12968 leader_election.cc:288] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5ba865194ad474081b82a7164092909 (127.4.197.129:36323), 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687)
I20260929 02:50:00.169215 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.161563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:42149 (local address 127.4.197.129:36323)
0929 02:50:00.161827 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:00.161833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:00.162887 (+ 1054us) server_negotiation.cc:407] Connection header received
0929 02:50:00.163129 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:00.163135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:00.163208 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:00.163300 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:00.163985 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.165149 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.167565 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.168350 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.168505 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.168580 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:00.168839 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 02:50:00.168960 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260929 02:50:00.169348 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.161886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36323 (local address 127.4.197.130:42149)
0929 02:50:00.162808 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:00.162823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:00.162998 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:00.163323 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:00.163328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:00.163360 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:00.163782 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:00.163794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.165279 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.165284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:00.167428 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:00.167438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.168549 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.168555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:00.168698 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.168722 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:50:00.168801 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:50:00.168920 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":21,"thread_start_us":625,"threads_started":1}
I20260929 02:50:00.169975 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.161802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:33375 (local address 127.4.197.131:36687)
0929 02:50:00.162007 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:00.162014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:00.162584 (+ 570us) server_negotiation.cc:407] Connection header received
0929 02:50:00.162902 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:00.162908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:00.163020 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:00.163131 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:00.163722 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.164804 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.167475 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.168786 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.168949 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.169016 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:00.169540 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 02:50:00.169684 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:50:00.170027 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377a85a8f77b44a4a610071b426e84b1" candidate_uuid: "456ab55e11a24deda4b6020b92d4afc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ba865194ad474081b82a7164092909" is_pre_election: true
I20260929 02:50:00.170295 12609 raft_consensus.cc:2454] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 456ab55e11a24deda4b6020b92d4afc0 in term 0.
I20260929 02:50:00.170639 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.161520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:36687 (local address 127.4.197.130:33375)
0929 02:50:00.162467 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:00.162488 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:00.162728 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:00.163137 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:00.163141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:00.163163 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:00.163540 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:00.163554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.165001 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.165009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:00.167309 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:00.167324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.169162 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.169168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:00.169355 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.169390 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:50:00.170146 (+ 756us) client_negotiation.cc:240] Negotiation successful
0929 02:50:00.170280 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":1,"thread_start_us":647,"threads_started":1}
I20260929 02:50:00.170769 12681 leader_election.cc:302] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456ab55e11a24deda4b6020b92d4afc0, f5ba865194ad474081b82a7164092909; no voters:
I20260929 02:50:00.171211 12968 raft_consensus.cc:2781] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:50:00.171312 12968 raft_consensus.cc:482] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:50:00.171267 12887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377a85a8f77b44a4a610071b426e84b1" candidate_uuid: "456ab55e11a24deda4b6020b92d4afc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29180caa0a1840dd856b88202e247789" is_pre_election: true
I20260929 02:50:00.171401 12968 raft_consensus.cc:3037] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:00.171525 12887 raft_consensus.cc:2454] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 456ab55e11a24deda4b6020b92d4afc0 in term 0.
I20260929 02:50:00.173566 12968 raft_consensus.cc:504] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.174105 12968 leader_election.cc:288] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [CANDIDATE]: Term 1 election: Requested vote from peers f5ba865194ad474081b82a7164092909 (127.4.197.129:36323), 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687)
I20260929 02:50:00.174566 12887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377a85a8f77b44a4a610071b426e84b1" candidate_uuid: "456ab55e11a24deda4b6020b92d4afc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29180caa0a1840dd856b88202e247789"
I20260929 02:50:00.174575 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377a85a8f77b44a4a610071b426e84b1" candidate_uuid: "456ab55e11a24deda4b6020b92d4afc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ba865194ad474081b82a7164092909"
I20260929 02:50:00.174767 12887 raft_consensus.cc:3037] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:00.174785 12609 raft_consensus.cc:3037] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:00.176975 12887 raft_consensus.cc:2454] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 456ab55e11a24deda4b6020b92d4afc0 in term 1.
I20260929 02:50:00.177321 12679 leader_election.cc:302] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29180caa0a1840dd856b88202e247789, 456ab55e11a24deda4b6020b92d4afc0; no voters:
I20260929 02:50:00.177563 12968 raft_consensus.cc:2781] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:00.177663 12609 raft_consensus.cc:2454] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 456ab55e11a24deda4b6020b92d4afc0 in term 1.
I20260929 02:50:00.178014 12968 raft_consensus.cc:686] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 LEADER]: Becoming Leader. State: Replica: 456ab55e11a24deda4b6020b92d4afc0, State: Running, Role: LEADER
I20260929 02:50:00.178448 12968 consensus_queue.cc:237] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.180580 13007 tablet_replica.cc:439] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: TxnStatusTablet state changed. Reason: New leader 456ab55e11a24deda4b6020b92d4afc0. Latest consensus state: current_term: 1 leader_uuid: "456ab55e11a24deda4b6020b92d4afc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } }
I20260929 02:50:00.180940 13007 tablet_replica.cc:442] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:50:00.181490 13009 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:50:00.181689 13009 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:50:00.181900 12452 catalog_manager.cc:5495] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130). New cstate: current_term: 1 leader_uuid: "456ab55e11a24deda4b6020b92d4afc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20260929 02:50:00.223603 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.217617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.4.197.130:41013)
0929 02:50:00.217806 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:00.217812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:00.218000 (+ 188us) server_negotiation.cc:407] Connection header received
0929 02:50:00.218306 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:00.218312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:00.218442 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:00.218580 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:00.219007 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.219801 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.221069 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.221583 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.221689 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.222192 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:00.222886 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:00.223123 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:00.223169 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 02:50:00.223280 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:50:00.223588 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.217720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:41013 (local address 127.0.0.1:40310)
0929 02:50:00.217917 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:00.217933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:00.218070 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:00.218607 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:00.218612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:00.218624 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:50:00.218845 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:00.218857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.219923 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.219929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:00.220929 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:00.220939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.221724 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.221729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:00.221916 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.221990 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:50:00.223026 (+ 1036us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:50:00.223052 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 02:50:00.223123 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 02:50:00.223282 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 02:50:00.227681 12609 raft_consensus.cc:1261] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Refusing update from remote peer 456ab55e11a24deda4b6020b92d4afc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:00.227869 12887 raft_consensus.cc:1261] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Refusing update from remote peer 456ab55e11a24deda4b6020b92d4afc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:00.228492 13007 consensus_queue.cc:1035] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:00.229053 13007 consensus_queue.cc:1035] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:00.243001 12969 tablet_replica.cc:439] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: TxnStatusTablet state changed. Reason: New leader 456ab55e11a24deda4b6020b92d4afc0. Latest consensus state: current_term: 1 leader_uuid: "456ab55e11a24deda4b6020b92d4afc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } }
I20260929 02:50:00.243394 12969 tablet_replica.cc:442] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:50:00.247542 13007 tablet_replica.cc:439] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "456ab55e11a24deda4b6020b92d4afc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } }
I20260929 02:50:00.247808 13007 tablet_replica.cc:442] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:50:00.248929 12977 mvcc.cc:205] Tried to move back new op lower bound from 7334503220125204480 to 7334503219934330880. Current Snapshot: MvccSnapshot[applied={T|T < 7334503220125204480}]
I20260929 02:50:00.248996 12983 tablet_replica.cc:439] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: TxnStatusTablet state changed. Reason: New leader 456ab55e11a24deda4b6020b92d4afc0. Latest consensus state: current_term: 1 leader_uuid: "456ab55e11a24deda4b6020b92d4afc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } }
I20260929 02:50:00.249261 12983 tablet_replica.cc:442] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:50:00.249207 12969 tablet_replica.cc:439] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "456ab55e11a24deda4b6020b92d4afc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } }
I20260929 02:50:00.249471 12969 tablet_replica.cc:442] T 377a85a8f77b44a4a610071b426e84b1 P f5ba865194ad474081b82a7164092909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:50:00.251912 13007 tablet_replica.cc:439] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "456ab55e11a24deda4b6020b92d4afc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } }
I20260929 02:50:00.252148 13007 tablet_replica.cc:442] T 377a85a8f77b44a4a610071b426e84b1 P 456ab55e11a24deda4b6020b92d4afc0: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:50:00.253645 12983 tablet_replica.cc:439] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "456ab55e11a24deda4b6020b92d4afc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } } }
I20260929 02:50:00.253894 12983 tablet_replica.cc:442] T 377a85a8f77b44a4a610071b426e84b1 P 29180caa0a1840dd856b88202e247789: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:50:00.268738 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.262577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.4.197.129:36323)
0929 02:50:00.262731 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:00.262737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:00.263432 (+ 695us) server_negotiation.cc:407] Connection header received
0929 02:50:00.263624 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:00.263628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:00.263712 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:00.263840 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:00.264274 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.265002 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.266464 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.266946 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.267057 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.267470 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:00.268088 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:00.268233 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:00.268289 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 02:50:00.268371 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:50:00.288578 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.279236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:40527 (local address 127.4.197.130:41013)
0929 02:50:00.279461 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:00.279468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:00.283555 (+ 4087us) server_negotiation.cc:407] Connection header received
0929 02:50:00.283751 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:00.283756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:00.283835 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:00.283910 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:00.284394 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.285145 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.286560 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.287147 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.287276 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.287805 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:00.287916 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:00.287994 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:00.288135 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 02:50:00.288225 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:50:00.289191 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.279128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:41013 (local address 127.4.197.129:40527)
0929 02:50:00.283445 (+ 4317us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:00.283462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:00.283618 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:00.283946 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:00.283950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:00.283962 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:50:00.284235 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:00.284244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.285267 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.285273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:00.286408 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:00.286422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.287345 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.287353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:00.287558 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.287632 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:50:00.288009 (+ 377us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:50:00.288029 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 02:50:00.288105 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 02:50:00.288255 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4133,"thread_start_us":3983,"threads_started":1}
I20260929 02:50:00.289500 12969 consensus_queue.cc:1035] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:00.297073 12969 consensus_queue.cc:1035] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:00.328881 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.322945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36323 (local address 127.4.197.130:51349)
0929 02:50:00.323468 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:00.323486 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:00.323629 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:00.323923 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:00.323928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:00.323939 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:50:00.324267 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:00.324276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.325349 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.325354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:00.326556 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:00.326567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.327495 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.327502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:00.327682 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.327758 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:50:00.328196 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:50:00.328207 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:50:00.328300 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 02:50:00.328442 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"spinlock_wait_cycles":7040,"thread_start_us":325,"threads_started":1}
I20260929 02:50:00.329818 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.322891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:51349 (local address 127.4.197.129:36323)
0929 02:50:00.323232 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:00.323238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:00.323565 (+ 327us) server_negotiation.cc:407] Connection header received
0929 02:50:00.323748 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:00.323753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:00.323830 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:00.323898 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:00.324426 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.325236 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.326678 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.327225 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.327341 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.327944 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:00.328067 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:00.328147 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:00.329470 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0929 02:50:00.329549 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260929 02:50:00.368185 12452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:00.368449 12452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:00.371847 12452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359: Sending DeleteTablet for 3 replicas of tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92
I20260929 02:50:00.372807 12867 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:00 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:36028
I20260929 02:50:00.373021 12728 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:00 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:40296
I20260929 02:50:00.373553 12588 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:00 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:36144
I20260929 02:50:00.381989 13029 tablet_replica.cc:330] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909: stopping tablet replica
I20260929 02:50:00.382189 13029 raft_consensus.cc:2229] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:50:00.382658 13029 raft_consensus.cc:2258] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:00.384893 13029 ts_tablet_manager.cc:1905] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:00.391090 13027 tablet_replica.cc:330] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789: stopping tablet replica
I20260929 02:50:00.391371 13027 raft_consensus.cc:2229] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:00.391659 13027 raft_consensus.cc:2258] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:00.392822 13027 ts_tablet_manager.cc:1905] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:00.396960 13028 tablet_replica.cc:330] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0: stopping tablet replica
I20260929 02:50:00.397289 13028 raft_consensus.cc:2229] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:00.397650 13028 raft_consensus.cc:2258] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:00.398703 13029 ts_tablet_manager.cc:1918] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:50:00.399571 13028 ts_tablet_manager.cc:1905] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:00.407099 13029 log.cc:1196] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/8db7b4b19aad4d9fb97be7b3b9bdbd92
I20260929 02:50:00.407716 13029 ts_tablet_manager.cc:1939] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P f5ba865194ad474081b82a7164092909: Deleting consensus metadata
I20260929 02:50:00.409219 13027 ts_tablet_manager.cc:1918] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:50:00.409235 12438 catalog_manager.cc:4841] TS f5ba865194ad474081b82a7164092909 (127.4.197.129:36323): tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92 (table test-table [id=9cd4b924348c4fc9a44d60f83f457c03]) successfully deleted
I20260929 02:50:00.409353 13027 log.cc:1196] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/8db7b4b19aad4d9fb97be7b3b9bdbd92
I20260929 02:50:00.409771 13027 ts_tablet_manager.cc:1939] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 29180caa0a1840dd856b88202e247789: Deleting consensus metadata
I20260929 02:50:00.410210 13028 ts_tablet_manager.cc:1918] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:50:00.410396 13028 log.cc:1196] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/8db7b4b19aad4d9fb97be7b3b9bdbd92
I20260929 02:50:00.410832 12436 catalog_manager.cc:4841] TS 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687): tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92 (table test-table [id=9cd4b924348c4fc9a44d60f83f457c03]) successfully deleted
I20260929 02:50:00.410905 13028 ts_tablet_manager.cc:1939] T 8db7b4b19aad4d9fb97be7b3b9bdbd92 P 456ab55e11a24deda4b6020b92d4afc0: Deleting consensus metadata
I20260929 02:50:00.412006 12439 catalog_manager.cc:4841] TS 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013): tablet 8db7b4b19aad4d9fb97be7b3b9bdbd92 (table test-table [id=9cd4b924348c4fc9a44d60f83f457c03]) successfully deleted
Sep 29 02:50:00 dist-test-slave-t6c1 krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:50:00 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:00 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:50:00.458611 12547 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:00.445092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.4.197.190:45593)
0929 02:50:00.445285 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:00.445291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:00.445848 (+ 557us) server_negotiation.cc:407] Connection header received
0929 02:50:00.446016 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:00.446022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:00.446112 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:00.446227 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:00.448120 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.449236 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.451078 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.451759 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.451887 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.458337 (+ 6450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42310: BlockingRecv error: recv got EOF from 127.0.0.1:42310 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 02:50:00 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM
I20260929 02:50:00.486161 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.464596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.4.197.190:45593)
0929 02:50:00.464786 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:00.464792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:00.467439 (+ 2647us) server_negotiation.cc:407] Connection header received
0929 02:50:00.467723 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:00.467730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:00.467815 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:00.467886 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:00.469661 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.470799 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.472755 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.473413 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.473510 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.478715 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:00.478722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:00.478727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:00.478783 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:00.482812 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:00.483870 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:00.483876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:00.483879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:00.484043 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:00.484523 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:00.484528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:00.484532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:00.484895 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:00.485123 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:00.485552 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 02:50:00.485803 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:50:00.490896 12452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:50:00.491647 12452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:50:00.499822 12588 tablet_service.cc:1467] Processing CreateTablet for tablet 1e39a76bc86b455e8483dde340a8b102 (DEFAULT_TABLE table=test-table [id=738ad92369234c2db92ab4a03c11a436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:00.500316 12588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e39a76bc86b455e8483dde340a8b102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:00.507233 12728 tablet_service.cc:1467] Processing CreateTablet for tablet 1e39a76bc86b455e8483dde340a8b102 (DEFAULT_TABLE table=test-table [id=738ad92369234c2db92ab4a03c11a436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:00.507694 12728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e39a76bc86b455e8483dde340a8b102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:00.511456 13042 tablet_bootstrap.cc:492] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909: Bootstrap starting.
I20260929 02:50:00.511960 12867 tablet_service.cc:1467] Processing CreateTablet for tablet 1e39a76bc86b455e8483dde340a8b102 (DEFAULT_TABLE table=test-table [id=738ad92369234c2db92ab4a03c11a436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:00.512339 12867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e39a76bc86b455e8483dde340a8b102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:00.514204 13042 tablet_bootstrap.cc:654] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:00.524895 13045 tablet_bootstrap.cc:492] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0: Bootstrap starting.
I20260929 02:50:00.527470 13045 tablet_bootstrap.cc:654] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:00.528115 13044 tablet_bootstrap.cc:492] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789: Bootstrap starting.
I20260929 02:50:00.530448 13044 tablet_bootstrap.cc:654] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:00.532066 13042 tablet_bootstrap.cc:492] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909: No bootstrap required, opened a new log
I20260929 02:50:00.532259 13042 ts_tablet_manager.cc:1397] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20260929 02:50:00.532573 13045 tablet_bootstrap.cc:492] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0: No bootstrap required, opened a new log
I20260929 02:50:00.532809 13045 ts_tablet_manager.cc:1397] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 02:50:00.533118 13042 raft_consensus.cc:348] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.533396 13042 raft_consensus.cc:374] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:00.533504 13042 raft_consensus.cc:729] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ba865194ad474081b82a7164092909, State: Initialized, Role: FOLLOWER
I20260929 02:50:00.533771 13042 consensus_queue.cc:260] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.533855 13045 raft_consensus.cc:348] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.534386 13042 ts_tablet_manager.cc:1428] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 02:50:00.534819 13045 raft_consensus.cc:374] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:00.534945 13045 raft_consensus.cc:729] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456ab55e11a24deda4b6020b92d4afc0, State: Initialized, Role: FOLLOWER
I20260929 02:50:00.535154 13045 consensus_queue.cc:260] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.535778 13045 ts_tablet_manager.cc:1428] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:50:00.538146 13044 tablet_bootstrap.cc:492] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789: No bootstrap required, opened a new log
I20260929 02:50:00.538298 13044 ts_tablet_manager.cc:1397] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260929 02:50:00.539119 13044 raft_consensus.cc:348] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.539364 13044 raft_consensus.cc:374] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:00.539456 13044 raft_consensus.cc:729] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29180caa0a1840dd856b88202e247789, State: Initialized, Role: FOLLOWER
I20260929 02:50:00.539691 13044 consensus_queue.cc:260] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.540839 13044 ts_tablet_manager.cc:1428] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:50:00.908308 13007 raft_consensus.cc:482] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:50:00.908519 13007 raft_consensus.cc:504] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.909171 13007 leader_election.cc:288] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5ba865194ad474081b82a7164092909 (127.4.197.129:36323), 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687)
I20260929 02:50:00.909799 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e39a76bc86b455e8483dde340a8b102" candidate_uuid: "456ab55e11a24deda4b6020b92d4afc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ba865194ad474081b82a7164092909" is_pre_election: true
I20260929 02:50:00.909871 12887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e39a76bc86b455e8483dde340a8b102" candidate_uuid: "456ab55e11a24deda4b6020b92d4afc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29180caa0a1840dd856b88202e247789" is_pre_election: true
I20260929 02:50:00.910077 12887 raft_consensus.cc:2454] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 456ab55e11a24deda4b6020b92d4afc0 in term 0.
I20260929 02:50:00.910105 12609 raft_consensus.cc:2454] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 456ab55e11a24deda4b6020b92d4afc0 in term 0.
I20260929 02:50:00.910557 12679 leader_election.cc:302] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29180caa0a1840dd856b88202e247789, 456ab55e11a24deda4b6020b92d4afc0, f5ba865194ad474081b82a7164092909; no voters:
I20260929 02:50:00.910842 13007 raft_consensus.cc:2781] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:50:00.910953 13007 raft_consensus.cc:482] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:50:00.911042 13007 raft_consensus.cc:3037] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:00.913285 13007 raft_consensus.cc:504] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.913992 13007 leader_election.cc:288] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [CANDIDATE]: Term 1 election: Requested vote from peers f5ba865194ad474081b82a7164092909 (127.4.197.129:36323), 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687)
I20260929 02:50:00.914502 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e39a76bc86b455e8483dde340a8b102" candidate_uuid: "456ab55e11a24deda4b6020b92d4afc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ba865194ad474081b82a7164092909"
I20260929 02:50:00.914482 12887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e39a76bc86b455e8483dde340a8b102" candidate_uuid: "456ab55e11a24deda4b6020b92d4afc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29180caa0a1840dd856b88202e247789"
I20260929 02:50:00.914714 12609 raft_consensus.cc:3037] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:00.914736 12887 raft_consensus.cc:3037] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:00.917593 12887 raft_consensus.cc:2454] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 456ab55e11a24deda4b6020b92d4afc0 in term 1.
I20260929 02:50:00.917613 12609 raft_consensus.cc:2454] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 456ab55e11a24deda4b6020b92d4afc0 in term 1.
I20260929 02:50:00.918027 12681 leader_election.cc:302] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29180caa0a1840dd856b88202e247789, 456ab55e11a24deda4b6020b92d4afc0, f5ba865194ad474081b82a7164092909; no voters:
I20260929 02:50:00.918330 13007 raft_consensus.cc:2781] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:00.918782 13007 raft_consensus.cc:686] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 LEADER]: Becoming Leader. State: Replica: 456ab55e11a24deda4b6020b92d4afc0, State: Running, Role: LEADER
I20260929 02:50:00.919075 13007 consensus_queue.cc:237] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:00.921495 12452 catalog_manager.cc:5495] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130). New cstate: current_term: 1 leader_uuid: "456ab55e11a24deda4b6020b92d4afc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20260929 02:50:00.997768 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:00.992071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.4.197.130:41013)
0929 02:50:00.992688 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:00.992694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:00.992713 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:50:00.992776 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:00.992783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:00.992850 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:00.992925 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:00.993375 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.994082 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.996104 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:00.996548 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:00.996642 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:00.997128 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:00.997217 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:00.997275 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:00.997406 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 02:50:00.997464 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":381,"threads_started":1}
I20260929 02:50:01.001209 12887 raft_consensus.cc:1261] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Refusing update from remote peer 456ab55e11a24deda4b6020b92d4afc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:01.001320 12609 raft_consensus.cc:1261] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Refusing update from remote peer 456ab55e11a24deda4b6020b92d4afc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:01.001883 13048 consensus_queue.cc:1035] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:01.002179 13007 consensus_queue.cc:1035] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:01.032498 12452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:01.032701 12452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:01.035593 12452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359: Sending DeleteTablet for 3 replicas of tablet 1e39a76bc86b455e8483dde340a8b102
I20260929 02:50:01.036321 12588 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e39a76bc86b455e8483dde340a8b102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:01 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:36144
I20260929 02:50:01.036705 12728 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e39a76bc86b455e8483dde340a8b102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:01 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:40296
I20260929 02:50:01.036769 12867 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e39a76bc86b455e8483dde340a8b102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:01 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:36028
I20260929 02:50:01.038276 13064 tablet_replica.cc:330] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789: stopping tablet replica
I20260929 02:50:01.038087 12452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 02:50:01.038479 13064 raft_consensus.cc:2229] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:01.038721 13064 raft_consensus.cc:2258] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 02:50:01.038784 12452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:50:01.039197 13063 tablet_replica.cc:330] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0: stopping tablet replica
I20260929 02:50:01.039422 13063 raft_consensus.cc:2229] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:50:01.039990 13063 raft_consensus.cc:2258] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:01.041062 13063 ts_tablet_manager.cc:1905] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:01.047871 12589 tablet_service.cc:1467] Processing CreateTablet for tablet d7aef1f28f004046a6a6b7c97fa7ec23 (DEFAULT_TABLE table=test-table [id=7d028d95b90d42fbb9717de875185ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:01.048374 12589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7aef1f28f004046a6a6b7c97fa7ec23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:01.048624 13062 tablet_replica.cc:330] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909: stopping tablet replica
I20260929 02:50:01.048759 13063 ts_tablet_manager.cc:1918] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:01.048815 13062 raft_consensus.cc:2229] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:01.048880 13063 log.cc:1196] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/1e39a76bc86b455e8483dde340a8b102
I20260929 02:50:01.049047 13062 raft_consensus.cc:2258] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:01.049628 13063 ts_tablet_manager.cc:1939] T 1e39a76bc86b455e8483dde340a8b102 P 456ab55e11a24deda4b6020b92d4afc0: Deleting consensus metadata
I20260929 02:50:01.049819 13062 ts_tablet_manager.cc:1905] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:01.050815 12439 catalog_manager.cc:4841] TS 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013): tablet 1e39a76bc86b455e8483dde340a8b102 (table test-table [id=738ad92369234c2db92ab4a03c11a436]) successfully deleted
I20260929 02:50:01.052021 12728 tablet_service.cc:1467] Processing CreateTablet for tablet d7aef1f28f004046a6a6b7c97fa7ec23 (DEFAULT_TABLE table=test-table [id=7d028d95b90d42fbb9717de875185ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:01.052169 12867 tablet_service.cc:1467] Processing CreateTablet for tablet d7aef1f28f004046a6a6b7c97fa7ec23 (DEFAULT_TABLE table=test-table [id=7d028d95b90d42fbb9717de875185ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:01.052495 12728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7aef1f28f004046a6a6b7c97fa7ec23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:01.052572 12867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7aef1f28f004046a6a6b7c97fa7ec23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:01.054553 13064 ts_tablet_manager.cc:1905] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:01.055872 13062 ts_tablet_manager.cc:1918] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:01.056066 13062 log.cc:1196] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/1e39a76bc86b455e8483dde340a8b102
I20260929 02:50:01.056522 13062 ts_tablet_manager.cc:1939] T 1e39a76bc86b455e8483dde340a8b102 P f5ba865194ad474081b82a7164092909: Deleting consensus metadata
I20260929 02:50:01.057646 12438 catalog_manager.cc:4841] TS f5ba865194ad474081b82a7164092909 (127.4.197.129:36323): tablet 1e39a76bc86b455e8483dde340a8b102 (table test-table [id=738ad92369234c2db92ab4a03c11a436]) successfully deleted
I20260929 02:50:01.064006 13064 ts_tablet_manager.cc:1918] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:01.064219 13064 log.cc:1196] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/1e39a76bc86b455e8483dde340a8b102
I20260929 02:50:01.065799 13064 ts_tablet_manager.cc:1939] T 1e39a76bc86b455e8483dde340a8b102 P 29180caa0a1840dd856b88202e247789: Deleting consensus metadata
I20260929 02:50:01.067001 12436 catalog_manager.cc:4841] TS 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687): tablet 1e39a76bc86b455e8483dde340a8b102 (table test-table [id=738ad92369234c2db92ab4a03c11a436]) successfully deleted
I20260929 02:50:01.067754 13067 tablet_bootstrap.cc:492] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0: Bootstrap starting.
I20260929 02:50:01.069861 13066 tablet_bootstrap.cc:492] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909: Bootstrap starting.
I20260929 02:50:01.071468 13065 tablet_bootstrap.cc:492] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789: Bootstrap starting.
I20260929 02:50:01.071866 13067 tablet_bootstrap.cc:654] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:01.072661 13066 tablet_bootstrap.cc:654] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:01.073307 13065 tablet_bootstrap.cc:654] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:01.080505 13067 tablet_bootstrap.cc:492] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0: No bootstrap required, opened a new log
I20260929 02:50:01.080734 13067 ts_tablet_manager.cc:1397] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260929 02:50:01.081487 13065 tablet_bootstrap.cc:492] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789: No bootstrap required, opened a new log
I20260929 02:50:01.081709 13065 ts_tablet_manager.cc:1397] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.008s
I20260929 02:50:01.081661 13067 raft_consensus.cc:348] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:01.081954 13067 raft_consensus.cc:374] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:01.082072 13067 raft_consensus.cc:729] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456ab55e11a24deda4b6020b92d4afc0, State: Initialized, Role: FOLLOWER
I20260929 02:50:01.082368 13067 consensus_queue.cc:260] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:01.082865 13065 raft_consensus.cc:348] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:01.083209 13065 raft_consensus.cc:374] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:01.083355 13065 raft_consensus.cc:729] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29180caa0a1840dd856b88202e247789, State: Initialized, Role: FOLLOWER
I20260929 02:50:01.083702 13065 consensus_queue.cc:260] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:01.084100 13067 ts_tablet_manager.cc:1428] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:50:01.084441 13065 ts_tablet_manager.cc:1428] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:50:01.084758 12983 raft_consensus.cc:482] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:50:01.084914 12983 raft_consensus.cc:504] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:01.085598 12983 leader_election.cc:288] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5ba865194ad474081b82a7164092909 (127.4.197.129:36323), 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013)
I20260929 02:50:01.085862 13066 tablet_bootstrap.cc:492] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909: No bootstrap required, opened a new log
I20260929 02:50:01.086031 13066 ts_tablet_manager.cc:1397] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260929 02:50:01.086299 12748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7aef1f28f004046a6a6b7c97fa7ec23" candidate_uuid: "29180caa0a1840dd856b88202e247789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456ab55e11a24deda4b6020b92d4afc0" is_pre_election: true
I20260929 02:50:01.086547 12748 raft_consensus.cc:2454] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29180caa0a1840dd856b88202e247789 in term 0.
I20260929 02:50:01.086854 13066 raft_consensus.cc:348] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:01.087173 13066 raft_consensus.cc:374] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:01.087302 13066 raft_consensus.cc:729] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ba865194ad474081b82a7164092909, State: Initialized, Role: FOLLOWER
I20260929 02:50:01.087416 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7aef1f28f004046a6a6b7c97fa7ec23" candidate_uuid: "29180caa0a1840dd856b88202e247789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ba865194ad474081b82a7164092909" is_pre_election: true
I20260929 02:50:01.087561 13066 consensus_queue.cc:260] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:01.088162 13066 ts_tablet_manager.cc:1428] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:50:01.088384 12821 leader_election.cc:302] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29180caa0a1840dd856b88202e247789, 456ab55e11a24deda4b6020b92d4afc0; no voters:
I20260929 02:50:01.088618 12609 raft_consensus.cc:2454] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29180caa0a1840dd856b88202e247789 in term 0.
I20260929 02:50:01.088688 12983 raft_consensus.cc:2781] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:50:01.088800 12983 raft_consensus.cc:482] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:50:01.088886 12983 raft_consensus.cc:3037] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:01.091725 12983 raft_consensus.cc:504] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:01.092310 12983 leader_election.cc:288] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [CANDIDATE]: Term 1 election: Requested vote from peers f5ba865194ad474081b82a7164092909 (127.4.197.129:36323), 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013)
I20260929 02:50:01.092659 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7aef1f28f004046a6a6b7c97fa7ec23" candidate_uuid: "29180caa0a1840dd856b88202e247789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ba865194ad474081b82a7164092909"
I20260929 02:50:01.092875 12609 raft_consensus.cc:3037] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:01.093044 12748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7aef1f28f004046a6a6b7c97fa7ec23" candidate_uuid: "29180caa0a1840dd856b88202e247789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456ab55e11a24deda4b6020b92d4afc0"
I20260929 02:50:01.093305 12748 raft_consensus.cc:3037] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:01.095872 12609 raft_consensus.cc:2454] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29180caa0a1840dd856b88202e247789 in term 1.
I20260929 02:50:01.096400 12748 raft_consensus.cc:2454] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29180caa0a1840dd856b88202e247789 in term 1.
I20260929 02:50:01.096598 12820 leader_election.cc:302] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29180caa0a1840dd856b88202e247789, f5ba865194ad474081b82a7164092909; no voters:
I20260929 02:50:01.097062 12983 raft_consensus.cc:2781] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:01.097461 12983 raft_consensus.cc:686] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 1 LEADER]: Becoming Leader. State: Replica: 29180caa0a1840dd856b88202e247789, State: Running, Role: LEADER
I20260929 02:50:01.097751 12983 consensus_queue.cc:237] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } }
I20260929 02:50:01.100657 12450 catalog_manager.cc:5495] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29180caa0a1840dd856b88202e247789 (127.4.197.131). New cstate: current_term: 1 leader_uuid: "29180caa0a1840dd856b88202e247789" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29180caa0a1840dd856b88202e247789" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36687 } health_report { overall_health: HEALTHY } } }
I20260929 02:50:01.125337 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:01.119759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.4.197.131:36687)
0929 02:50:01.120286 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:01.120290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:01.120308 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:50:01.120358 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:01.120362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:01.120418 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:01.120493 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:01.120896 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.121742 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.123643 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.124086 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.124176 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:01.124621 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:01.124725 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:01.124787 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:01.124916 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 02:50:01.124977 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":288,"threads_started":1}
I20260929 02:50:01.140115 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:01.134370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:34669 (local address 127.4.197.130:41013)
0929 02:50:01.134546 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:01.134552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:01.135014 (+ 462us) server_negotiation.cc:407] Connection header received
0929 02:50:01.135202 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:01.135214 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:01.135279 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:01.135356 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:01.135874 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.136619 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.138564 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.138990 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.139080 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:01.139548 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:01.139642 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:01.139707 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:01.139833 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 02:50:01.139898 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:50:01.140345 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:01.134193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:41013 (local address 127.4.197.131:34669)
0929 02:50:01.134902 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:01.134940 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:01.135090 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:01.135385 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:01.135391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:01.135405 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:50:01.135717 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:01.135728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.136750 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.136756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:01.138433 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:01.138448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.139142 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.139148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:01.139326 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:01.139397 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:50:01.139727 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:50:01.139746 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 02:50:01.139822 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 02:50:01.139962 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":416,"threads_started":1}
I20260929 02:50:01.148598 12609 raft_consensus.cc:1261] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Refusing update from remote peer 29180caa0a1840dd856b88202e247789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:01.148594 12748 raft_consensus.cc:1261] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Refusing update from remote peer 29180caa0a1840dd856b88202e247789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:01.149096 12983 consensus_queue.cc:1035] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ba865194ad474081b82a7164092909" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:01.149386 13071 consensus_queue.cc:1035] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "456ab55e11a24deda4b6020b92d4afc0" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:01.180720 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:01.171968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:32943 (local address 127.4.197.131:36687)
0929 02:50:01.172110 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:01.172116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:01.172859 (+ 743us) server_negotiation.cc:407] Connection header received
0929 02:50:01.173108 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:01.173113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:01.173191 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:01.173264 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:01.173985 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.174985 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.178825 (+ 3840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.179428 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.179499 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:01.180008 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:01.180143 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:01.180218 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:01.180363 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 02:50:01.180436 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:50:01.180951 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:01.172025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:36687 (local address 127.4.197.130:32943)
0929 02:50:01.172764 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:01.172783 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:01.172956 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:01.173468 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:01.173473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:01.173484 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:50:01.173781 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:01.173794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.176954 (+ 3160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.176959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:01.178666 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:01.178682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.179642 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.179648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:01.179823 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:01.179863 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:50:01.180249 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:50:01.180260 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:50:01.180336 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 02:50:01.180487 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":435,"threads_started":1}
I20260929 02:50:01.217273 12450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:01.217555 12450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:01.220327 12450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d13112991eb4af68bd0676cfe326359: Sending DeleteTablet for 3 replicas of tablet d7aef1f28f004046a6a6b7c97fa7ec23
I20260929 02:50:01.220976 12589 tablet_service.cc:1514] Processing DeleteTablet for tablet d7aef1f28f004046a6a6b7c97fa7ec23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:01 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:36144
I20260929 02:50:01.221006 12728 tablet_service.cc:1514] Processing DeleteTablet for tablet d7aef1f28f004046a6a6b7c97fa7ec23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:01 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:40296
I20260929 02:50:01.221249 13062 tablet_replica.cc:330] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909: stopping tablet replica
I20260929 02:50:01.221254 12867 tablet_service.cc:1514] Processing DeleteTablet for tablet d7aef1f28f004046a6a6b7c97fa7ec23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:01 UTC) from {username='oryx', principal='oryx/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:36028
I20260929 02:50:01.221437 13062 raft_consensus.cc:2229] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:01.221513 13064 tablet_replica.cc:330] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789: stopping tablet replica
I20260929 02:50:01.221622 13062 raft_consensus.cc:2258] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:01.221683 13064 raft_consensus.cc:2229] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:50:01.221890 13063 tablet_replica.cc:330] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0: stopping tablet replica
I20260929 02:50:01.221980 13064 raft_consensus.cc:2258] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:01.222074 13063 raft_consensus.cc:2229] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:01.222254 13063 raft_consensus.cc:2258] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:01.223181 13064 ts_tablet_manager.cc:1905] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:01.223457 13063 ts_tablet_manager.cc:1905] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:01.227530 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 12523
I20260929 02:50:01.229607 13064 ts_tablet_manager.cc:1918] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:50:01.229871 13064 log.cc:1196] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/d7aef1f28f004046a6a6b7c97fa7ec23
I20260929 02:50:01.229923 13063 ts_tablet_manager.cc:1918] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:50:01.230414 13064 ts_tablet_manager.cc:1939] T d7aef1f28f004046a6a6b7c97fa7ec23 P 29180caa0a1840dd856b88202e247789: Deleting consensus metadata
I20260929 02:50:01.230662 13062 ts_tablet_manager.cc:1905] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:01.231559 12436 catalog_manager.cc:4841] TS 29180caa0a1840dd856b88202e247789 (127.4.197.131:36687): tablet d7aef1f28f004046a6a6b7c97fa7ec23 (table test-table [id=7d028d95b90d42fbb9717de875185ac7]) successfully deleted
I20260929 02:50:01.235415 13062 ts_tablet_manager.cc:1918] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:50:01.235632 13062 log.cc:1196] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/d7aef1f28f004046a6a6b7c97fa7ec23
I20260929 02:50:01.236080 13062 ts_tablet_manager.cc:1939] T d7aef1f28f004046a6a6b7c97fa7ec23 P f5ba865194ad474081b82a7164092909: Deleting consensus metadata
I20260929 02:50:01.237047 13063 log.cc:1196] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/d7aef1f28f004046a6a6b7c97fa7ec23
I20260929 02:50:01.237191 12438 catalog_manager.cc:4841] TS f5ba865194ad474081b82a7164092909 (127.4.197.129:36323): tablet d7aef1f28f004046a6a6b7c97fa7ec23 (table test-table [id=7d028d95b90d42fbb9717de875185ac7]) successfully deleted
I20260929 02:50:01.237545 13063 ts_tablet_manager.cc:1939] T d7aef1f28f004046a6a6b7c97fa7ec23 P 456ab55e11a24deda4b6020b92d4afc0: Deleting consensus metadata
I20260929 02:50:01.238410 12439 catalog_manager.cc:4841] TS 456ab55e11a24deda4b6020b92d4afc0 (127.4.197.130:41013): tablet d7aef1f28f004046a6a6b7c97fa7ec23 (table test-table [id=7d028d95b90d42fbb9717de875185ac7]) successfully deleted
Sep 29 02:50:01 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.129@KRBTEST.COM
I20260929 02:50:01.253300 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:01.229092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.4.197.129:36323)
0929 02:50:01.236612 (+ 7520us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:01.236618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:01.238742 (+ 2124us) server_negotiation.cc:407] Connection header received
0929 02:50:01.238898 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:01.238904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:01.239025 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:01.239136 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:01.240216 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.241344 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.242667 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.243343 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.243467 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:01.247602 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:01.247623 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:01.247639 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:01.247681 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:01.250412 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:01.251299 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:01.251307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:01.251309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:01.251436 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:01.251807 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:01.251810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:01.251812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:01.252163 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:01.252443 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:01.252764 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 02:50:01.252981 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7382,"thread_start_us":7260,"threads_started":1}
I20260929 02:50:01.253996 12654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:01.403162 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 12523
I20260929 02:50:01.422216 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 12663
Sep 29 02:50:01 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.130@KRBTEST.COM
I20260929 02:50:01.437899 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:01.423382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.4.197.130:41013)
0929 02:50:01.423591 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:01.423597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:01.423611 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:50:01.423665 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:01.423669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:01.423728 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:01.423817 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:01.425014 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.425785 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.427060 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.427515 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.427636 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:01.431854 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:01.431868 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:01.431884 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:01.431921 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:01.435123 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:01.435791 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:01.435804 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:01.435808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:01.435984 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:01.436337 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:01.436342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:01.436346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:01.436727 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:01.437019 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:01.437332 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 02:50:01.437537 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:50:01.438612 12793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:01.598414 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 12663
I20260929 02:50:01.618877 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 12802
Sep 29 02:50:01 dist-test-slave-t6c1 krb5kdc[12407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.197.131@KRBTEST.COM
I20260929 02:50:01.632949 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:01.619621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.4.197.131:36687)
0929 02:50:01.619792 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:01.619796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:01.619813 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:50:01.619954 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:01.619957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:01.620017 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:01.620094 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:01.621246 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.621946 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.623349 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:01.623766 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:01.623860 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:01.628016 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:01.628030 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:01.628045 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:01.628082 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:01.630648 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:01.631349 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:01.631356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:01.631358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:01.631490 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:01.631781 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:01.631784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:01.631786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:01.632128 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:01.632345 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:01.632531 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 02:50:01.632683 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:50:01.633621 12932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:01.756328 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 12802
I20260929 02:50:01.774128 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 12420
I20260929 02:50:01.775318 12492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:01.887681 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 12420
2026-09-29T02:50:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6649 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:50:01 dist-test-slave-t6c1 krb5kdc[13092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:50:01 dist-test-slave-t6c1 krb5kdc[13092](info): set up 2 sockets
Sep 29 02:50:01 dist-test-slave-t6c1 krb5kdc[13092](info): commencing operation
krb5kdc: starting...
W20260929 02:50:02.891728 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:50:03 dist-test-slave-t6c1 krb5kdc[13092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:50:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:50:03.092614 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:40083
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:40083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:03.405661 13105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:03.405943 13105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:03.406070 13105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:03.413023 13105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:03.413115 13105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:03.413187 13105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:03.413257 13105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:03.413321 13105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:03.422771 13105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:40083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:40083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:03.424003 13105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:03.425261 13105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:03.436094 13110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:03.436057 13113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:03.436020 13111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:03.436898 13105 server_base.cc:1034] running on GCE node
I20260929 02:50:03.437573 13105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:03.438942 13105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:03.440158 13105 hybrid_clock.cc:648] HybridClock initialized: now 1790650203440109 us; error 40 us; skew 500 ppm
I20260929 02:50:03.440465 13105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T02:50:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1625 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:50:03 dist-test-slave-t6c1 krb5kdc[13119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:50:03 dist-test-slave-t6c1 krb5kdc[13119](info): set up 2 sockets
Sep 29 02:50:03 dist-test-slave-t6c1 krb5kdc[13119](info): commencing operation
krb5kdc: starting...
W20260929 02:50:04.515576 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.005s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:50:04 dist-test-slave-t6c1 krb5kdc[13119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:50:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:50:04.724375 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:42207
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:42207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:05.043056 13132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:05.043354 13132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:05.043486 13132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:05.050987 13132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:05.051106 13132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:05.051178 13132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:05.051254 13132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:05.051316 13132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:05.061388 13132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:42207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:42207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:05.062814 13132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:05.064189 13132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:05.076289 13137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:05.076310 13140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:05.076341 13138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:05.077174 13132 server_base.cc:1034] running on GCE node
I20260929 02:50:05.077997 13132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:05.079581 13132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:05.080801 13132 hybrid_clock.cc:648] HybridClock initialized: now 1790650205080745 us; error 41 us; skew 500 ppm
I20260929 02:50:05.081156 13132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:05 dist-test-slave-t6c1 krb5kdc[13119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650205, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:05.084259 13132 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:50:05.085820 13132 webserver.cc:460] Webserver started at http://127.4.197.190:44369/ using document root <none> and password file <none>
I20260929 02:50:05.086519 13132 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:05.086633 13132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:05.086993 13132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:05.090015 13132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "77c0a28e95084f03b88fcbd0e72e1c91"
format_stamp: "Formatted at 2026-09-29 02:50:05 on dist-test-slave-t6c1"
server_key: "17187f5a18eb0e271193edbca33f95d6"
server_key_iv: "689cb0d0bd7b4670749302a6e267a362"
server_key_version: "encryptionkey@0"
I20260929 02:50:05.090804 13132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "77c0a28e95084f03b88fcbd0e72e1c91"
format_stamp: "Formatted at 2026-09-29 02:50:05 on dist-test-slave-t6c1"
server_key: "17187f5a18eb0e271193edbca33f95d6"
server_key_iv: "689cb0d0bd7b4670749302a6e267a362"
server_key_version: "encryptionkey@0"
I20260929 02:50:05.095829 13132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:50:05.099164 13147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:05.100451 13132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:50:05.100605 13132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "77c0a28e95084f03b88fcbd0e72e1c91"
format_stamp: "Formatted at 2026-09-29 02:50:05 on dist-test-slave-t6c1"
server_key: "17187f5a18eb0e271193edbca33f95d6"
server_key_iv: "689cb0d0bd7b4670749302a6e267a362"
server_key_version: "encryptionkey@0"
I20260929 02:50:05.100780 13132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:05.117033 13132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:05.120698 13132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:05.120930 13132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:05.138377 13132 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:42207
I20260929 02:50:05.138384 13198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:42207 every 8 connection(s)
I20260929 02:50:05.139880 13132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:50:05.143733 13199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:05.144109 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 13132
I20260929 02:50:05.144325 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:50:05.145103 4886 external_mini_cluster.cc:1428] Setting key 3d32557032c1240d3bb9c7968915bffc
I20260929 02:50:05.157994 13199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91: Bootstrap starting.
Sep 29 02:50:05 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:50:05.161795 13199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:05.163098 13199 log.cc:824] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:05.166309 13199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91: No bootstrap required, opened a new log
I20260929 02:50:05.169544 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:05.147299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.4.197.190:42207)
0929 02:50:05.148074 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:05.148086 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:05.148127 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:50:05.150004 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:05.150033 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:05.150496 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:05.151034 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:05.152816 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:05.154203 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:05.156157 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:05.156934 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:05.157083 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:05.161729 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:05.161738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:05.161754 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:05.161802 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:05.165244 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:05.166009 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:05.166016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:05.166023 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:05.166204 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:05.166598 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:05.166602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:05.166604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:05.167213 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:05.167497 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:05.167964 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 02:50:05.168432 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":340,"threads_started":1}
I20260929 02:50:05.174353 13199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } }
I20260929 02:50:05.174746 13199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:05.174844 13199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c0a28e95084f03b88fcbd0e72e1c91, State: Initialized, Role: FOLLOWER
I20260929 02:50:05.175402 13199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } }
I20260929 02:50:05.175621 13199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:50:05.175776 13199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:50:05.175926 13199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:05.178570 13199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } }
I20260929 02:50:05.179157 13199 leader_election.cc:302] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c0a28e95084f03b88fcbd0e72e1c91; no voters:
I20260929 02:50:05.179802 13199 leader_election.cc:288] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:50:05.179948 13204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:05.180960 13204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 1 LEADER]: Becoming Leader. State: Replica: 77c0a28e95084f03b88fcbd0e72e1c91, State: Running, Role: LEADER
I20260929 02:50:05.181505 13204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } }
I20260929 02:50:05.182022 13199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:50:05.184502 13206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c0a28e95084f03b88fcbd0e72e1c91. Latest consensus state: current_term: 1 leader_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } } }
I20260929 02:50:05.184949 13206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:05.185537 13205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } } }
I20260929 02:50:05.185838 13205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:05.192286 13213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:50:05.197501 13213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:50:05.205711 13213 catalog_manager.cc:1347] Generated new cluster ID: 9810ac295a5d407a99207ca997ce0398
I20260929 02:50:05.205817 13213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:50:05.219215 13213 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:50:05.220448 13213 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:50:05.229220 13213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91: Generated new TSK 0
I20260929 02:50:05.230029 13213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:50:05.290316 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42207
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:05.610248 13225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:05.610519 13225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:05.610610 13225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:05.618711 13225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:05.618811 13225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:05.618953 13225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:50:05.628618 13225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:05.629837 13225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:05.631070 13225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:05.642839 13231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:05.642913 13233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:05.642946 13230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:05.643810 13225 server_base.cc:1034] running on GCE node
I20260929 02:50:05.644323 13225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:05.645171 13225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:05.646368 13225 hybrid_clock.cc:648] HybridClock initialized: now 1790650205646321 us; error 45 us; skew 500 ppm
I20260929 02:50:05.646644 13225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:05 dist-test-slave-t6c1 krb5kdc[13119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650205, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:05.649678 13225 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:50:05.651022 13225 webserver.cc:460] Webserver started at http://127.4.197.129:42991/ using document root <none> and password file <none>
I20260929 02:50:05.651638 13225 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:05.651728 13225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:05.651968 13225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:05.654816 13225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "b70203505ed049d9b6f30813287df627"
format_stamp: "Formatted at 2026-09-29 02:50:05 on dist-test-slave-t6c1"
server_key: "2a308e56a24ab376460e7b7e3b7ca5bd"
server_key_iv: "881820e286851e4021a85e0333fbe530"
server_key_version: "encryptionkey@0"
I20260929 02:50:05.655625 13225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "b70203505ed049d9b6f30813287df627"
format_stamp: "Formatted at 2026-09-29 02:50:05 on dist-test-slave-t6c1"
server_key: "2a308e56a24ab376460e7b7e3b7ca5bd"
server_key_iv: "881820e286851e4021a85e0333fbe530"
server_key_version: "encryptionkey@0"
I20260929 02:50:05.660873 13225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:50:05.664393 13240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:05.665867 13225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:50:05.666009 13225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "b70203505ed049d9b6f30813287df627"
format_stamp: "Formatted at 2026-09-29 02:50:05 on dist-test-slave-t6c1"
server_key: "2a308e56a24ab376460e7b7e3b7ca5bd"
server_key_iv: "881820e286851e4021a85e0333fbe530"
server_key_version: "encryptionkey@0"
I20260929 02:50:05.666191 13225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:05.681535 13225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:05.685690 13225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:05.685953 13225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:05.687001 13225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:05.688257 13225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:05.688331 13225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:05.688423 13225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:05.688467 13225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:05.717185 13225 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:37531
I20260929 02:50:05.717234 13352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:37531 every 8 connection(s)
I20260929 02:50:05.718410 13225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:50:05.719795 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 13225
I20260929 02:50:05.720008 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:50:05.720811 4886 external_mini_cluster.cc:1428] Setting key 001aa47c8860995c6c24515411568f97
Sep 29 02:50:05 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650205, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:50:05.745402 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:05.721089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:33085 (local address 127.4.197.190:42207)
0929 02:50:05.721562 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:05.721569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:05.722979 (+ 1410us) server_negotiation.cc:407] Connection header received
0929 02:50:05.725139 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:05.725144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:05.725223 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:05.725317 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:05.727375 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:05.728109 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:05.730331 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:05.731030 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:05.732286 (+ 1256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:05.738537 (+ 6251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:05.738546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:05.738552 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:05.738617 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:05.741640 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:05.742721 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:05.742727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:05.742732 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:05.742878 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:05.743357 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:05.743362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:05.743366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:05.743749 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:05.743889 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:05.744828 (+ 939us) server_negotiation.cc:299] Negotiation successful
0929 02:50:05.745048 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260929 02:50:05.746587 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:05.721602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:33085)
0929 02:50:05.722344 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:05.722393 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:05.724811 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:05.725590 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:05.725601 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:05.726375 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:05.727194 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:05.727220 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:05.728268 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:05.728273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:05.730168 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:05.730183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:05.731500 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:05.731508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:05.731849 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:05.732976 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:05.733038 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:05.737895 (+ 4857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:05.741866 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:05.741876 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:05.741896 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:05.742573 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:05.743019 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:05.743025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:05.743029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:05.743234 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:05.743887 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:05.743896 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:05.744339 (+ 443us) client_negotiation.cc:769] Sending connection context
0929 02:50:05.744667 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 02:50:05.745222 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":281,"threads_started":1}
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:50:05.748967 13353 heartbeater.cc:347] Connected to a master server at 127.4.197.190:42207
I20260929 02:50:05.749320 13353 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:05.750216 13353 heartbeater.cc:511] Master 127.4.197.190:42207 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
I20260929 02:50:05.752511 13164 ts_manager.cc:194] Registered new tserver with Master: b70203505ed049d9b6f30813287df627 (127.4.197.129:37531)
I20260929 02:50:05.754175 13164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:33085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:50:05.777768 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42207
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:06.089016 13359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:06.089339 13359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:06.089471 13359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:06.096789 13359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:06.096913 13359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:06.097064 13359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:50:06.107079 13359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:06.108532 13359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:06.109957 13359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:06.123342 13365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:06.123378 13367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:06.123409 13364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:06.124091 13359 server_base.cc:1034] running on GCE node
I20260929 02:50:06.124790 13359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:06.125725 13359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:06.126935 13359 hybrid_clock.cc:648] HybridClock initialized: now 1790650206126859 us; error 51 us; skew 500 ppm
I20260929 02:50:06.127251 13359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:06 dist-test-slave-t6c1 krb5kdc[13119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650206, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:06.130141 13359 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:50:06.131510 13359 webserver.cc:460] Webserver started at http://127.4.197.130:40785/ using document root <none> and password file <none>
I20260929 02:50:06.132158 13359 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:06.132270 13359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:06.132540 13359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:06.135349 13359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "f44f73920ec74d2b8107276983666fe8"
format_stamp: "Formatted at 2026-09-29 02:50:06 on dist-test-slave-t6c1"
server_key: "0dfd40cee479ef6adf873e9d4412800c"
server_key_iv: "56a54b13f3b2eb32ee91834e5706a8fe"
server_key_version: "encryptionkey@0"
I20260929 02:50:06.136108 13359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "f44f73920ec74d2b8107276983666fe8"
format_stamp: "Formatted at 2026-09-29 02:50:06 on dist-test-slave-t6c1"
server_key: "0dfd40cee479ef6adf873e9d4412800c"
server_key_iv: "56a54b13f3b2eb32ee91834e5706a8fe"
server_key_version: "encryptionkey@0"
I20260929 02:50:06.141537 13359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:50:06.145143 13374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:06.146657 13359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:50:06.146811 13359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "f44f73920ec74d2b8107276983666fe8"
format_stamp: "Formatted at 2026-09-29 02:50:06 on dist-test-slave-t6c1"
server_key: "0dfd40cee479ef6adf873e9d4412800c"
server_key_iv: "56a54b13f3b2eb32ee91834e5706a8fe"
server_key_version: "encryptionkey@0"
I20260929 02:50:06.147045 13359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:06.173049 13359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:06.177132 13359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:06.177397 13359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:06.178352 13359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:06.179745 13359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:06.179819 13359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:06.179934 13359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:06.179986 13359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:06.209834 13359 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:42681
I20260929 02:50:06.209874 13486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:42681 every 8 connection(s)
I20260929 02:50:06.211123 13359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:50:06.217809 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 13359
I20260929 02:50:06.218000 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:50:06.218546 4886 external_mini_cluster.cc:1428] Setting key 27d76ae4ce53c540f5ad14b76e38aa26
Sep 29 02:50:06 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650206, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:50:06.231189 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:06.213540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:59465 (local address 127.4.197.190:42207)
0929 02:50:06.213700 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:06.213704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:06.214876 (+ 1172us) server_negotiation.cc:407] Connection header received
0929 02:50:06.216702 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:06.216708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:06.216788 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:06.216876 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:06.218723 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.219609 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.221577 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.221963 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.222056 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:06.225801 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:06.225806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:06.225809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:06.225841 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:06.228370 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:06.229007 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:06.229012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:06.229015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:06.229141 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:06.229538 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:06.229545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:06.229547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:06.229816 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:06.229947 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:06.230609 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 02:50:06.230841 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:50:06.232112 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:06.214012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:59465)
0929 02:50:06.214697 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:06.214735 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:06.216434 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:06.217110 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:06.217118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:06.217714 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:06.218506 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:06.218532 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.219803 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.219807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:06.221429 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:06.221443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.222161 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.222168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:06.222439 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:06.223174 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:06.223200 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:06.225594 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:06.228495 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:06.228504 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:06.228519 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:06.228899 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:06.229249 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:06.229253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:06.229255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:06.229423 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:06.229939 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:06.229947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:06.230261 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 02:50:06.230501 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 02:50:06.231025 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":251,"threads_started":1}
I20260929 02:50:06.233513 13487 heartbeater.cc:347] Connected to a master server at 127.4.197.190:42207
I20260929 02:50:06.233831 13487 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:06.234486 13487 heartbeater.cc:511] Master 127.4.197.190:42207 requested a full tablet report, sending...
I20260929 02:50:06.235913 13164 ts_manager.cc:194] Registered new tserver with Master: f44f73920ec74d2b8107276983666fe8 (127.4.197.130:42681)
I20260929 02:50:06.236671 13164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:59465
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:50:06.264182 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42207
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:06.577435 13492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:06.577754 13492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:06.577888 13492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:06.585325 13492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:06.585450 13492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:06.585608 13492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:50:06.595476 13492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:06.596720 13492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:06.597961 13492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:06.610152 13497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:06.610204 13500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:06.610245 13498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:06.611208 13492 server_base.cc:1034] running on GCE node
I20260929 02:50:06.611714 13492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:06.612509 13492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:06.613695 13492 hybrid_clock.cc:648] HybridClock initialized: now 1790650206613647 us; error 42 us; skew 500 ppm
I20260929 02:50:06.613967 13492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:06 dist-test-slave-t6c1 krb5kdc[13119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650206, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:06.616783 13492 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:50:06.618183 13492 webserver.cc:460] Webserver started at http://127.4.197.131:41371/ using document root <none> and password file <none>
I20260929 02:50:06.618963 13492 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:06.619093 13492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:06.619421 13492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:06.622412 13492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "2d1f6eef36f0423bb3cad35fa15e2980"
format_stamp: "Formatted at 2026-09-29 02:50:06 on dist-test-slave-t6c1"
server_key: "35fbb7982471c220897ebd8c76fec8fe"
server_key_iv: "19ddebdd8c642ef1ad1819f1a374c443"
server_key_version: "encryptionkey@0"
I20260929 02:50:06.623268 13492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "2d1f6eef36f0423bb3cad35fa15e2980"
format_stamp: "Formatted at 2026-09-29 02:50:06 on dist-test-slave-t6c1"
server_key: "35fbb7982471c220897ebd8c76fec8fe"
server_key_iv: "19ddebdd8c642ef1ad1819f1a374c443"
server_key_version: "encryptionkey@0"
I20260929 02:50:06.628453 13492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:50:06.632046 13507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:06.633534 13492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:50:06.633702 13492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "2d1f6eef36f0423bb3cad35fa15e2980"
format_stamp: "Formatted at 2026-09-29 02:50:06 on dist-test-slave-t6c1"
server_key: "35fbb7982471c220897ebd8c76fec8fe"
server_key_iv: "19ddebdd8c642ef1ad1819f1a374c443"
server_key_version: "encryptionkey@0"
I20260929 02:50:06.633905 13492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:06.648954 13492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:06.653051 13492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:06.653303 13492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:06.654228 13492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:06.655615 13492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:06.655685 13492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:06.655777 13492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:06.655830 13492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:06.683517 13492 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:35709
I20260929 02:50:06.683554 13619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:35709 every 8 connection(s)
I20260929 02:50:06.684734 13492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:50:06.693302 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 13492
I20260929 02:50:06.693456 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:50:06.693881 4886 external_mini_cluster.cc:1428] Setting key 1fd19db20e5be80aa35497a65cd4e2d4
Sep 29 02:50:06 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650206, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:50:06.704391 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:06.687359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:49467 (local address 127.4.197.190:42207)
0929 02:50:06.687576 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:06.687581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:06.688771 (+ 1190us) server_negotiation.cc:407] Connection header received
0929 02:50:06.690637 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:06.690643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:06.690762 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:06.690860 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:06.692399 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.693187 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.695074 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.695687 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.695797 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:06.699400 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:06.699404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:06.699408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:06.699440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:06.701690 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:06.702353 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:06.702357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:06.702359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:06.702473 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:06.702823 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:06.702825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:06.702828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:06.703168 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:06.703278 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:06.703967 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 02:50:06.704142 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:50:06.705281 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:06.687888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:49467)
0929 02:50:06.688612 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:06.688650 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:06.690362 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:06.691135 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:06.691143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:06.691662 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:06.692223 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:06.692242 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.693350 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.693354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:06.694903 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:06.694947 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.695901 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.695906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:06.696205 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:06.697003 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:06.697039 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:06.699191 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:06.701831 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:06.701838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:06.701850 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:06.702240 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:06.702547 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:06.702550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:06.702552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:06.702743 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:06.703261 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:06.703267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:06.703590 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 02:50:06.703842 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 02:50:06.704212 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":268,"threads_started":1}
I20260929 02:50:06.706609 13620 heartbeater.cc:347] Connected to a master server at 127.4.197.190:42207
I20260929 02:50:06.706904 13620 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:06.707706 13620 heartbeater.cc:511] Master 127.4.197.190:42207 requested a full tablet report, sending...
I20260929 02:50:06.709383 13163 ts_manager.cc:194] Registered new tserver with Master: 2d1f6eef36f0423bb3cad35fa15e2980 (127.4.197.131:35709)
I20260929 02:50:06.710232 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:49467
I20260929 02:50:06.721215 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.129@KRBTEST.COM" created.
I20260929 02:50:06.757730 13353 heartbeater.cc:503] Master 127.4.197.190:42207 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab.
Entry for principal oryx/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab.
WARNING: no policy specified for oryx/127.4.197.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab.
Entry for principal oryx/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab.
WARNING: no policy specified for oryx/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab.
Entry for principal oryx/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab.
I20260929 02:50:06.841584 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13225
Sep 29 02:50:06 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:50:06.857182 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:06.842623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.4.197.129:37531)
0929 02:50:06.843447 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:06.843454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:06.843479 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:50:06.843550 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:06.843557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:06.843765 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:06.843905 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:06.845108 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.846087 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.847297 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.847853 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.847946 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:06.851839 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:06.851853 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:06.851873 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:06.851920 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:06.854522 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:06.855269 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:06.855276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:06.855279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:06.855416 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:06.855775 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:06.855780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:06.855782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:06.856152 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:06.856400 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:06.856628 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 02:50:06.856871 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":507,"threads_started":1}
I20260929 02:50:06.858429 13348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:06.939019 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 13225
I20260929 02:50:06.952730 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13359
Sep 29 02:50:06 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:50:06.967772 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:06.953548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.4.197.130:42681)
0929 02:50:06.954089 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:06.954096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:06.954122 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:50:06.954181 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:06.954187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:06.954399 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:06.954528 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:06.955789 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.956777 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.958002 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:06.958539 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:06.958636 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:06.962731 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:06.962744 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:06.962758 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:06.962796 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:06.965427 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:06.966051 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:06.966058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:06.966060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:06.966175 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:06.966426 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:06.966428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:06.966430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:06.966751 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:06.967010 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:06.967215 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 02:50:06.967411 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":275,"threads_started":1}
I20260929 02:50:06.968952 13482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:07.049571 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 13359
I20260929 02:50:07.061615 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13492
Sep 29 02:50:07 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:50:07.077508 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:07.062597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.4.197.131:35709)
0929 02:50:07.063264 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:07.063271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:07.063301 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:50:07.063361 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:07.063367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:07.063590 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:07.063721 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:07.065023 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:07.066057 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:07.067372 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:07.067946 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:07.068047 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:07.072164 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:07.072176 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:07.072190 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:07.072240 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:07.075162 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:07.075834 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:07.075840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:07.075842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:07.075959 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:07.076261 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:07.076264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:07.076266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:07.076586 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:07.076811 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:07.077018 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 02:50:07.077221 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":360,"threads_started":1}
I20260929 02:50:07.078780 13615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:07.159344 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 13492
I20260929 02:50:07.171057 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13132
I20260929 02:50:07.172160 13194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:07.273244 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 13132
I20260929 02:50:07.289740 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:42207
--webserver_interface=127.4.197.190
--webserver_port=44369
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:42207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:07.606542 13637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:07.606865 13637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:07.607028 13637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:07.614336 13637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:07.614467 13637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:07.614549 13637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:07.614634 13637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:07.614694 13637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:07.625869 13637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:42207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:42207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=44369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:07.627270 13637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:07.628572 13637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:07.639974 13642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:07.639937 13643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:07.639896 13645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:07.640662 13637 server_base.cc:1034] running on GCE node
I20260929 02:50:07.641441 13637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:07.642853 13637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:07.644070 13637 hybrid_clock.cc:648] HybridClock initialized: now 1790650207644006 us; error 57 us; skew 500 ppm
I20260929 02:50:07.644403 13637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:07 dist-test-slave-t6c1 krb5kdc[13119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650207, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:07.647404 13637 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:50:07.648849 13637 webserver.cc:460] Webserver started at http://127.4.197.190:44369/ using document root <none> and password file <none>
I20260929 02:50:07.649531 13637 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:07.649650 13637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:07.655263 13637 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:50:07.658247 13652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:07.659720 13637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:50:07.659883 13637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "77c0a28e95084f03b88fcbd0e72e1c91"
format_stamp: "Formatted at 2026-09-29 02:50:05 on dist-test-slave-t6c1"
server_key: "17187f5a18eb0e271193edbca33f95d6"
server_key_iv: "689cb0d0bd7b4670749302a6e267a362"
server_key_version: "encryptionkey@0"
I20260929 02:50:07.660859 13637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:07.679207 13637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:07.683460 13637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:07.683727 13637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:07.700570 13637 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:42207
I20260929 02:50:07.700590 13703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:42207 every 8 connection(s)
I20260929 02:50:07.701969 13637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:50:07.708415 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 13637
I20260929 02:50:07.708765 13704 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:50:07.709378 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:37531
--local_ip_for_outbound_sockets=127.4.197.129
--tserver_master_addrs=127.4.197.190:42207
--webserver_port=42991
--webserver_interface=127.4.197.129
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.197.129
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:50:07.713035 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91: Bootstrap starting.
I20260929 02:50:07.732822 13704 log.cc:824] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:07.743428 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:50:07.743927 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91: Bootstrap complete.
I20260929 02:50:07.752692 13704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } }
I20260929 02:50:07.753536 13704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c0a28e95084f03b88fcbd0e72e1c91, State: Initialized, Role: FOLLOWER
I20260929 02:50:07.754148 13704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } }
I20260929 02:50:07.754407 13704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:50:07.754509 13704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:50:07.754675 13704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:50:07.758389 13704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } }
I20260929 02:50:07.758888 13704 leader_election.cc:302] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c0a28e95084f03b88fcbd0e72e1c91; no voters:
I20260929 02:50:07.759526 13704 leader_election.cc:288] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:50:07.759740 13708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:50:07.760685 13708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [term 2 LEADER]: Becoming Leader. State: Replica: 77c0a28e95084f03b88fcbd0e72e1c91, State: Running, Role: LEADER
I20260929 02:50:07.761511 13704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:50:07.761418 13708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } }
I20260929 02:50:07.765237 13709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } } }
I20260929 02:50:07.765203 13710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c0a28e95084f03b88fcbd0e72e1c91. Latest consensus state: current_term: 2 leader_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c0a28e95084f03b88fcbd0e72e1c91" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42207 } } }
I20260929 02:50:07.765636 13709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:07.765640 13710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:07.772028 13718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:50:07.777669 13718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:50:07.779174 13718 catalog_manager.cc:1259] Loaded cluster ID: 9810ac295a5d407a99207ca997ce0398
I20260929 02:50:07.779362 13718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:50:07.783754 13718 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:50:07.785459 13718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77c0a28e95084f03b88fcbd0e72e1c91: Loaded TSK: 0
I20260929 02:50:07.787261 13718 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:50:08.053781 13706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:08.054162 13706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:08.054309 13706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:08.061607 13706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:08.061736 13706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:08.061906 13706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:50:08.072345 13706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.129.keytab
--principal=oryx/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:37531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=42991
--tserver_master_addrs=127.4.197.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:08.073930 13706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:08.075575 13706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:08.091464 13730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:08.091487 13731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:08.091686 13733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:08.092941 13706 server_base.cc:1034] running on GCE node
I20260929 02:50:08.093528 13706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:08.094455 13706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:08.095649 13706 hybrid_clock.cc:648] HybridClock initialized: now 1790650208095589 us; error 52 us; skew 500 ppm
I20260929 02:50:08.095961 13706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:08 dist-test-slave-t6c1 krb5kdc[13119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650208, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:08.099058 13706 init.cc:377] Logged in from keytab as oryx/127.4.197.129@KRBTEST.COM (short username oryx)
I20260929 02:50:08.100564 13706 webserver.cc:460] Webserver started at http://127.4.197.129:42991/ using document root <none> and password file <none>
I20260929 02:50:08.101222 13706 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:08.101343 13706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:08.106460 13706 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:50:08.108938 13740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:08.110191 13706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:50:08.110334 13706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "b70203505ed049d9b6f30813287df627"
format_stamp: "Formatted at 2026-09-29 02:50:05 on dist-test-slave-t6c1"
server_key: "2a308e56a24ab376460e7b7e3b7ca5bd"
server_key_iv: "881820e286851e4021a85e0333fbe530"
server_key_version: "encryptionkey@0"
I20260929 02:50:08.111245 13706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:08.122732 13706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:08.126559 13706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:08.126788 13706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:08.127696 13706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:08.128887 13706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:08.128968 13706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:08.129061 13706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:08.129135 13706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:08.158285 13706 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:37531
I20260929 02:50:08.158332 13852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:37531 every 8 connection(s)
I20260929 02:50:08.159806 13706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:50:08.161692 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 13706
I20260929 02:50:08.162362 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:42681
--local_ip_for_outbound_sockets=127.4.197.130
--tserver_master_addrs=127.4.197.190:42207
--webserver_port=40785
--webserver_interface=127.4.197.130
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.197.130
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:50:08 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:08 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:50:08.478055 13857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:08.478349 13857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:08.478463 13857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:08.485901 13857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:08.486008 13857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:08.486151 13857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:50:08.495978 13857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.130.keytab
--principal=oryx/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:42681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=40785
--tserver_master_addrs=127.4.197.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:08.497246 13857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:08.498524 13857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:08.513201 13864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:08.513315 13863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:08.513840 13866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:08.514328 13857 server_base.cc:1034] running on GCE node
I20260929 02:50:08.514897 13857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:08.515861 13857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:08.517045 13857 hybrid_clock.cc:648] HybridClock initialized: now 1790650208516986 us; error 64 us; skew 500 ppm
I20260929 02:50:08.517328 13857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:08 dist-test-slave-t6c1 krb5kdc[13119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650208, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:08.523275 13857 init.cc:377] Logged in from keytab as oryx/127.4.197.130@KRBTEST.COM (short username oryx)
I20260929 02:50:08.524765 13857 webserver.cc:460] Webserver started at http://127.4.197.130:40785/ using document root <none> and password file <none>
I20260929 02:50:08.525425 13857 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:08.525537 13857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:08.530828 13857 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:50:08.533586 13873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:08.535003 13857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:50:08.535144 13857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "f44f73920ec74d2b8107276983666fe8"
format_stamp: "Formatted at 2026-09-29 02:50:06 on dist-test-slave-t6c1"
server_key: "0dfd40cee479ef6adf873e9d4412800c"
server_key_iv: "56a54b13f3b2eb32ee91834e5706a8fe"
server_key_version: "encryptionkey@0"
I20260929 02:50:08.536069 13857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:08.548413 13857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:08.552716 13857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:08.552970 13857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:08.554070 13857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:08.555552 13857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:08.555646 13857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:08.555744 13857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:08.555804 13857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:08.594827 13857 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:42681
I20260929 02:50:08.594875 13985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:42681 every 8 connection(s)
I20260929 02:50:08.596268 13857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:50:08.604636 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 13857
I20260929 02:50:08.605522 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:35709
--local_ip_for_outbound_sockets=127.4.197.131
--tserver_master_addrs=127.4.197.190:42207
--webserver_port=41371
--webserver_interface=127.4.197.131
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.197.131
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:50:08 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:08 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:50:08.937402 13990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:08.937709 13990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:08.937870 13990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:08.945488 13990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:08.945624 13990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:08.945806 13990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:50:08.955762 13990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/krb5kdc/oryx_127.4.197.131.keytab
--principal=oryx/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:35709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=41371
--tserver_master_addrs=127.4.197.190:42207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:08.957126 13990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:08.958467 13990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:08.974490 13996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:08.980921 13997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:08.981070 13990 server_base.cc:1034] running on GCE node
W20260929 02:50:08.981438 13999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:08.982118 13990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:08.983160 13990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:08.984376 13990 hybrid_clock.cc:648] HybridClock initialized: now 1790650208984322 us; error 48 us; skew 500 ppm
I20260929 02:50:08.984692 13990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:08 dist-test-slave-t6c1 krb5kdc[13119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650208, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:08.987756 13990 init.cc:377] Logged in from keytab as oryx/127.4.197.131@KRBTEST.COM (short username oryx)
I20260929 02:50:08.989478 13990 webserver.cc:460] Webserver started at http://127.4.197.131:41371/ using document root <none> and password file <none>
I20260929 02:50:08.990202 13990 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:08.990334 13990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:08.995826 13990 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:50:08.998461 14006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:08.999858 13990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:50:09.000005 13990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "2d1f6eef36f0423bb3cad35fa15e2980"
format_stamp: "Formatted at 2026-09-29 02:50:06 on dist-test-slave-t6c1"
server_key: "35fbb7982471c220897ebd8c76fec8fe"
server_key_iv: "19ddebdd8c642ef1ad1819f1a374c443"
server_key_version: "encryptionkey@0"
I20260929 02:50:09.000996 13990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:09.026813 13990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:09.031009 13990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:09.031272 13990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:09.032362 13990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:09.033953 13990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:09.034047 13990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:09.034157 13990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:09.034224 13990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:09.078200 13990 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:35709
I20260929 02:50:09.078248 14118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:35709 every 8 connection(s)
I20260929 02:50:09.079659 13990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:50:09.084169 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 13990
Sep 29 02:50:09 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:09 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:09.117517 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:09.086004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.4.197.190:42207)
0929 02:50:09.096461 (+ 10457us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:09.096468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:09.096492 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:50:09.096558 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:09.096563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:09.096667 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:09.096801 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:09.099035 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.100448 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.102473 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.103180 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.103301 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.107563 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:09.107590 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:09.107615 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:09.107670 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:09.111830 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:09.112776 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:09.112786 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:09.112797 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:09.113023 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:09.113504 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:09.113509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:09.113513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:09.114150 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:09.114480 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:09.115890 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0929 02:50:09.116381 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10297,"thread_start_us":10179,"threads_started":1}
W20260929 02:50:09.383383 13856 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.209s user 0.003s sys 0.009s
W20260929 02:50:09.385344 13855 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:08.163043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:60595 (local address 127.4.197.190:42207)
0929 02:50:08.163773 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:08.163788 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:08.164374 (+ 586us) server_negotiation.cc:407] Connection header received
0929 02:50:08.166762 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:08.166781 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:08.167286 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:08.167673 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:08.169474 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:08.171309 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:08.172799 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:08.173721 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:08.173879 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.384940 (+1211061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:60595: BlockingRecv error: recv got EOF from 127.4.197.129:60595 (error 108)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":246,"threads_started":1}
I20260929 02:50:09.386050 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:08.163171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:60595)
0929 02:50:08.164187 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:08.164234 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:08.166177 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:08.167838 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:08.167849 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:08.168530 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:08.169282 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:08.169307 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:08.171460 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:08.171466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:08.172672 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:08.172684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:08.173906 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:08.173910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:08.174088 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:08.174687 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:08.174716 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:09.384458 (+1209742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":555,"threads_started":1}
W20260929 02:50:09.387348 13853 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:42207 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.197.190:42207: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Sep 29 02:50:09 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:09 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:09.400247 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:09.388209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:49775)
0929 02:50:09.388512 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:09.388535 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:09.388663 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:09.389045 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:09.389051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:09.389573 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:09.389808 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.389821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.391507 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.391513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.393208 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.393224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.394231 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.394238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.394435 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.395181 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:09.395209 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:09.399904 (+ 4695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163}
W20260929 02:50:09.400452 13855 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:09.388375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:49775 (local address 127.4.197.190:42207)
0929 02:50:09.388567 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:09.388573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:09.388595 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:50:09.388799 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:09.388805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:09.388889 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:09.389005 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:09.390045 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.391353 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.393356 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.394034 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.394173 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.400180 (+ 6007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:49775: BlockingRecv error: recv got EOF from 127.4.197.129:49775 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 02:50:09 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:09 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:09.413203 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:09.401375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:49055)
0929 02:50:09.401682 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:09.401703 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:09.401822 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:09.402176 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:09.402181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:09.402617 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:09.402831 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.402843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.404489 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.404495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.406116 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.406130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.407331 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.407338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.407540 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.408205 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:09.408231 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:09.412892 (+ 4661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182}
W20260929 02:50:09.413424 13855 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:09.401488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:49055 (local address 127.4.197.190:42207)
0929 02:50:09.401673 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:09.401680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:09.401758 (+ 78us) server_negotiation.cc:407] Connection header received
0929 02:50:09.401935 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:09.401940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:09.402021 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:09.402137 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:09.402991 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.404339 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.406255 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.406977 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.407109 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.413143 (+ 6034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:49055: BlockingRecv error: recv got EOF from 127.4.197.129:49055 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 02:50:09.413705 13853 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:50:09.753418 13989 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.145s user 0.003s sys 0.009s
W20260929 02:50:09.754673 13988 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:08.599068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:49631 (local address 127.4.197.190:42207)
0929 02:50:08.599478 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:08.599482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:08.600893 (+ 1411us) server_negotiation.cc:407] Connection header received
0929 02:50:08.602644 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:08.602649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:08.602733 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:08.602826 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:08.604262 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:08.605016 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:08.606568 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:08.607013 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:08.607146 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.754338 (+1147192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:49631: BlockingRecv error: recv got EOF from 127.4.197.130:49631 (error 108)
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":219,"threads_started":1}
I20260929 02:50:09.755293 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:08.599927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:49631)
0929 02:50:08.600743 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:08.600786 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:08.602388 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:08.603088 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:08.603095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:08.603565 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:08.604095 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:08.604112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:08.605150 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:08.605154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:08.606438 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:08.606449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:08.607377 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:08.607385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:08.607619 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:08.608265 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:08.608294 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:09.754132 (+1145838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":327,"threads_started":1}
W20260929 02:50:09.756397 13986 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:42207 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.197.190:42207: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Sep 29 02:50:09 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:09 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:50:09.768893 13988 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:09.757106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:35175 (local address 127.4.197.190:42207)
0929 02:50:09.757259 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:09.757263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:09.757347 (+ 84us) server_negotiation.cc:407] Connection header received
0929 02:50:09.757572 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:09.757575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:09.757645 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:09.757722 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:09.758771 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.759558 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.761649 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.762175 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.762305 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.768721 (+ 6416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:35175: BlockingRecv error: recv got EOF from 127.4.197.130:35175 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:50:09.768962 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:09.756971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:35175)
0929 02:50:09.757246 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:09.757271 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:09.757436 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:09.757784 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:09.757790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:09.758348 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:09.758624 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.758639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.759689 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.759695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.761490 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.761509 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.762389 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.762397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.762626 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.763718 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:09.763751 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:09.768554 (+ 4803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
Sep 29 02:50:09 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:09 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:09.779238 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:09.769627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:32935)
0929 02:50:09.769785 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:09.769801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:09.769895 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:09.770469 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:09.770473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:09.770741 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:09.770883 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.770890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.772490 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.772494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.773417 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.773428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.774480 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.774487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.774680 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.775251 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:09.775274 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:09.779023 (+ 3749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 02:50:09.779654 13986 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:50:09.779742 13988 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:09.769907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:32935 (local address 127.4.197.190:42207)
0929 02:50:09.770105 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:09.770112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:09.770134 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:50:09.770208 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:09.770215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:09.770316 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:09.770456 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:09.771014 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.772339 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.773542 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.774294 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.774441 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.779377 (+ 4936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:32935: BlockingRecv error: recv got EOF from 127.4.197.130:32935 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 02:50:10.154822 14121 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.047s user 0.004s sys 0.013s
W20260929 02:50:10.155879 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:09.085641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:55763 (local address 127.4.197.190:42207)
0929 02:50:09.096143 (+ 10502us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:09.096151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:09.096177 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:50:09.096256 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:09.096263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:09.096352 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:09.096479 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:09.101161 (+ 4682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.102444 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.105107 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.105813 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.105927 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.155688 (+1049761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:55763: BlockingRecv error: recv got EOF from 127.4.197.131:55763 (error 108)
Metrics: {"server-negotiator.queue_time_us":10364,"thread_start_us":10225,"threads_started":1}
I20260929 02:50:10.156445 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:09.083298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:55763)
0929 02:50:09.091666 (+ 8368us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:09.091736 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:09.094484 (+ 2748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:09.097385 (+ 2901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:09.097401 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:09.098238 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:09.100928 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.100961 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.102628 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.102634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.104898 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.104915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.106258 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.106265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.106581 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.107312 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:09.107345 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:10.155537 (+1048192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7933,"thread_start_us":7676,"threads_started":1}
W20260929 02:50:10.157234 14119 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:42207 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.197.190:42207: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Sep 29 02:50:10 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:10 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:10.166216 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.157789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:58751)
0929 02:50:10.157972 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:10.157992 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:10.158127 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:10.158399 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:10.158404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:10.158841 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:10.159067 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.159079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.160026 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.160032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.161209 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.161222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.161862 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.161867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.161986 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.162503 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:10.162525 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:10.165946 (+ 3421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 02:50:10.166334 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:10.157846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:58751 (local address 127.4.197.190:42207)
0929 02:50:10.157994 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.157998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.158078 (+ 80us) server_negotiation.cc:407] Connection header received
0929 02:50:10.158222 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.158225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.158284 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.158357 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:10.159187 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.159905 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.161321 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.161705 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.161807 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.166138 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:58751: BlockingRecv error: recv got EOF from 127.4.197.131:58751 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 02:50:10 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:10 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:10.173856 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.166881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:59577)
0929 02:50:10.167031 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:10.167049 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:10.167151 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:10.167420 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:10.167423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:10.167714 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:10.167849 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.167856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.168749 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.168752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.169672 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.169682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.170327 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.170331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.170448 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.170972 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:10.170993 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:10.173692 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 02:50:10.174079 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:10.166984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:59577 (local address 127.4.197.190:42207)
0929 02:50:10.167194 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.167198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.167214 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:50:10.167248 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.167251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.167313 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.167383 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:10.167966 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.168648 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.169793 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.170225 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.170310 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.173857 (+ 3547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:59577: BlockingRecv error: recv got EOF from 127.4.197.131:59577 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
W20260929 02:50:10.174101 14119 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 02:50:10 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:10 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:10.422617 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.414224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:39849)
0929 02:50:10.414749 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:10.414769 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:10.414890 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:10.415242 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:10.415247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:10.415673 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:10.415855 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.415866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.416810 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.416815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.417750 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.417761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.418457 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.418463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.418614 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.419219 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:10.419238 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:10.422387 (+ 3149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":261,"threads_started":1}
W20260929 02:50:10.422739 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:10.414305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:39849 (local address 127.4.197.190:42207)
0929 02:50:10.414508 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.414514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.414852 (+ 338us) server_negotiation.cc:407] Connection header received
0929 02:50:10.415019 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.415024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.415112 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.415193 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:10.415982 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.416674 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.417879 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.418292 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.418396 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.422557 (+ 4161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:39849: BlockingRecv error: recv got EOF from 127.4.197.129:39849 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 02:50:10 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:10 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:10.789294 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.780267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:33199)
0929 02:50:10.780984 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:10.781002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:10.781159 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:10.781496 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:10.781502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:10.781958 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:10.782116 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.782127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.783236 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.783240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.784185 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.784198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.784927 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.784933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.785088 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.785674 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:10.785691 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:10.788994 (+ 3303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":306,"threads_started":1}
W20260929 02:50:10.789443 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:10.780342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:33199 (local address 127.4.197.190:42207)
0929 02:50:10.780550 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.780556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.781117 (+ 561us) server_negotiation.cc:407] Connection header received
0929 02:50:10.781289 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.781295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.781366 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.781447 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:10.782269 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.783091 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.784336 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.784760 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.784886 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.789204 (+ 4318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:33199: BlockingRecv error: recv got EOF from 127.4.197.130:33199 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 02:50:11 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:11 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:11.183416 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.174793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:41597)
0929 02:50:11.175375 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:11.175394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:11.175525 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:11.175847 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:11.175852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:11.176296 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:11.176448 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:11.176457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.177444 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.177450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:11.178410 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:11.178423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.179206 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.179213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:11.179384 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.179959 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:11.179977 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:11.183188 (+ 3211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":331,"threads_started":1}
W20260929 02:50:11.183590 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:11.174882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:41597 (local address 127.4.197.190:42207)
0929 02:50:11.175095 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:11.175101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:11.175474 (+ 373us) server_negotiation.cc:407] Connection header received
0929 02:50:11.175639 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:11.175642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:11.175717 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:11.175798 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:11.176588 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.177306 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.178549 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.179024 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.179129 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.183391 (+ 4262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:41597: BlockingRecv error: recv got EOF from 127.4.197.131:41597 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 02:50:11 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:11 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:11.432243 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.423593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:54911)
0929 02:50:11.424065 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:11.424082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:11.424238 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:11.424560 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:11.424565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:11.425016 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:11.425248 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:11.425260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.426382 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.426387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:11.427355 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:11.427367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.428042 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.428048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:11.428215 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.428771 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:11.428788 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:11.431874 (+ 3086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":266,"threads_started":1}
W20260929 02:50:11.432350 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:11.423664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:54911 (local address 127.4.197.190:42207)
0929 02:50:11.423850 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:11.423856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:11.424179 (+ 323us) server_negotiation.cc:407] Connection header received
0929 02:50:11.424360 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:11.424364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:11.424429 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:11.424510 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:11.425388 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.426256 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.427490 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.427903 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.428007 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.432101 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:54911: BlockingRecv error: recv got EOF from 127.4.197.129:54911 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 02:50:11 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:11 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:11.798882 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.790142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:38613)
0929 02:50:11.790672 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:11.790688 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:11.790840 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:11.791166 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:11.791171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:11.791566 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:11.791746 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:11.791755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.792718 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.792724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:11.793641 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:11.793651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.794305 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.794310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:11.794480 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.795053 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:11.795070 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:11.798594 (+ 3524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":286,"threads_started":1}
W20260929 02:50:11.799024 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:11.790246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:38613 (local address 127.4.197.190:42207)
0929 02:50:11.790460 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:11.790466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:11.790771 (+ 305us) server_negotiation.cc:407] Connection header received
0929 02:50:11.790979 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:11.790983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:11.791043 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:11.791129 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:11.791882 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.792587 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.793767 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.794166 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.794274 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.798777 (+ 4503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:38613: BlockingRecv error: recv got EOF from 127.4.197.130:38613 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 02:50:12 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:12 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:12.193567 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:12.184334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:55563)
0929 02:50:12.184818 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:12.184834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:12.184948 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:12.185255 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:12.185258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:12.185656 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:12.185858 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:12.185866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.186945 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.186950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:12.187871 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:12.187883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.188547 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.188551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:12.188668 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:12.189246 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:12.189262 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:12.193242 (+ 3980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":277,"threads_started":1}
W20260929 02:50:12.193780 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:12.184479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:55563 (local address 127.4.197.190:42207)
0929 02:50:12.184708 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:12.184714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:12.184911 (+ 197us) server_negotiation.cc:407] Connection header received
0929 02:50:12.185053 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:12.185057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:12.185140 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:12.185229 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:12.186012 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.186812 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.188016 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.188428 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.188526 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:12.193489 (+ 4963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:55563: BlockingRecv error: recv got EOF from 127.4.197.131:55563 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 02:50:12 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:12 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:12.441470 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:12.433169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:56557)
0929 02:50:12.433607 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:12.433624 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:12.433743 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:12.434013 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:12.434016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:12.434361 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:12.434540 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:12.434549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.435551 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.435555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:12.436460 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:12.436471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.437145 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.437148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:12.437278 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:12.437819 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:12.437835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:12.441226 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
W20260929 02:50:12.441696 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:12.433258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:56557 (local address 127.4.197.190:42207)
0929 02:50:12.433484 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:12.433490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:12.433697 (+ 207us) server_negotiation.cc:407] Connection header received
0929 02:50:12.433847 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:12.433851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:12.433913 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:12.433983 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:12.434649 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.435455 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.436590 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.437033 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.437125 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:12.441447 (+ 4322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:56557: BlockingRecv error: recv got EOF from 127.4.197.129:56557 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 02:50:12 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:12 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:12.807956 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:12.799808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:57497)
0929 02:50:12.800281 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:12.800297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:12.800410 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:12.800708 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:12.800712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:12.801037 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:12.801200 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:12.801210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.802226 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.802229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:12.803198 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:12.803210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.803948 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.803953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:12.804118 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:12.804708 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:12.804725 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:12.807647 (+ 2922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
W20260929 02:50:12.808080 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:12.799937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:57497 (local address 127.4.197.190:42207)
0929 02:50:12.800120 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:12.800124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:12.800363 (+ 239us) server_negotiation.cc:407] Connection header received
0929 02:50:12.800520 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:12.800523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:12.800587 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:12.800662 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:12.801350 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.802089 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.803357 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:12.803812 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:12.803920 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:12.807857 (+ 3937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:57497: BlockingRecv error: recv got EOF from 127.4.197.130:57497 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 02:50:13 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:13 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:13.203665 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:13.194619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:36157)
0929 02:50:13.195125 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:13.195140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:13.195274 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:13.195629 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:13.195633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:13.196044 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:13.196230 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:13.196240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.197273 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.197278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:13.198291 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:13.198307 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.199200 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.199206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:13.199378 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:13.200087 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:13.200112 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:13.203437 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":275,"threads_started":1}
W20260929 02:50:13.203840 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:13.194674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:36157 (local address 127.4.197.190:42207)
0929 02:50:13.194947 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:13.194954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:13.195212 (+ 258us) server_negotiation.cc:407] Connection header received
0929 02:50:13.195369 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:13.195373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:13.195449 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:13.195536 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:13.196386 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.197145 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.198436 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.199032 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.199168 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:13.203642 (+ 4474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:36157: BlockingRecv error: recv got EOF from 127.4.197.131:36157 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 02:50:13 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:13 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:13.451280 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:13.442555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:35259)
0929 02:50:13.443070 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:13.443086 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:13.443210 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:13.443512 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:13.443516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:13.443907 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:13.444069 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:13.444077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.445262 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.445267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:13.446308 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:13.446325 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.447075 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.447079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:13.447223 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:13.447811 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:13.447827 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:13.451014 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":273,"threads_started":1}
W20260929 02:50:13.451443 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:13.442633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:35259 (local address 127.4.197.190:42207)
0929 02:50:13.442813 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:13.442819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:13.443165 (+ 346us) server_negotiation.cc:407] Connection header received
0929 02:50:13.443318 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:13.443322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:13.443382 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:13.443459 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:13.444225 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.445106 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.446476 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.446907 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.447046 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:13.451217 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:35259: BlockingRecv error: recv got EOF from 127.4.197.129:35259 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 02:50:13 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:13 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:13.821655 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:13.809062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:37849)
0929 02:50:13.809607 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:13.809623 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:13.809758 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:13.810073 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:13.810078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:13.810532 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:13.810727 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:13.810740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.811983 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.811986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:13.812944 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:13.812955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.813630 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.813634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:13.813759 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:13.818072 (+ 4313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:13.818093 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:13.821325 (+ 3232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
W20260929 02:50:13.821813 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:13.809155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:37849 (local address 127.4.197.190:42207)
0929 02:50:13.809360 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:13.809367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:13.809715 (+ 348us) server_negotiation.cc:407] Connection header received
0929 02:50:13.809880 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:13.809884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:13.809944 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:13.810024 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:13.810874 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.811858 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.813059 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:13.813474 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:13.813571 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:13.821586 (+ 8015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:37849: BlockingRecv error: recv got EOF from 127.4.197.130:37849 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 02:50:14 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:14 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:14.213161 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:14.204542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:53757)
0929 02:50:14.205072 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:14.205088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:14.205214 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:14.205526 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:14.205529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:14.205885 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:14.206043 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:14.206052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.207076 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.207079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:14.208116 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:14.208127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.208882 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.208889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:14.209094 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:14.209708 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:14.209727 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:14.212890 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":307,"threads_started":1}
W20260929 02:50:14.213258 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:14.204732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:53757 (local address 127.4.197.190:42207)
0929 02:50:14.204898 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:14.204902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:14.205158 (+ 256us) server_negotiation.cc:407] Connection header received
0929 02:50:14.205308 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:14.205312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:14.205404 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:14.205503 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:14.206176 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.206962 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.208248 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.208725 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.208854 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:14.213054 (+ 4200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:53757: BlockingRecv error: recv got EOF from 127.4.197.131:53757 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 02:50:14 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:14 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:14.460992 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:14.452238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:55005)
0929 02:50:14.452744 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:14.452761 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:14.452882 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:14.453144 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:14.453147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:14.453576 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:14.453765 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:14.453774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.454761 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.454766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:14.455755 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:14.455768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.456471 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.456476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:14.456639 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:14.457219 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:14.457237 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:14.460739 (+ 3502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
W20260929 02:50:14.461130 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:14.452318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:55005 (local address 127.4.197.190:42207)
0929 02:50:14.452549 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:14.452555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:14.452839 (+ 284us) server_negotiation.cc:407] Connection header received
0929 02:50:14.452971 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:14.452974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:14.453035 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:14.453114 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:14.453896 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.454611 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.455894 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.456320 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.456420 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:14.460926 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:55005: BlockingRecv error: recv got EOF from 127.4.197.129:55005 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 02:50:14 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:14 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:14.831357 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:14.822688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:52885)
0929 02:50:14.823193 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:14.823211 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:14.823342 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:14.823617 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:14.823620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:14.824000 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:14.824161 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:14.824170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.825181 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.825186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:14.826292 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:14.826305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.827184 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.827190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:14.827343 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:14.827860 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:14.827876 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:14.831126 (+ 3250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":403,"threads_started":1}
W20260929 02:50:14.831522 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:14.822892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:52885 (local address 127.4.197.190:42207)
0929 02:50:14.823084 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:14.823090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:14.823290 (+ 200us) server_negotiation.cc:407] Connection header received
0929 02:50:14.823438 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:14.823441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:14.823503 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:14.823586 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:14.824295 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.825037 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.826418 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:14.827040 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:14.827171 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:14.831310 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:52885: BlockingRecv error: recv got EOF from 127.4.197.130:52885 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 02:50:15 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:15 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:15.222349 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:15.214012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:52777)
0929 02:50:15.214741 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:15.214760 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:15.214899 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:15.215290 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:15.215296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:15.215721 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:15.215864 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:15.215873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.216838 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.216842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:15.217733 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:15.217743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.218349 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.218353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:15.218465 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:15.219008 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:15.219026 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:15.222102 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":248,"threads_started":1}
W20260929 02:50:15.222604 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:15.214109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:52777 (local address 127.4.197.190:42207)
0929 02:50:15.214305 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:15.214312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:15.214850 (+ 538us) server_negotiation.cc:407] Connection header received
0929 02:50:15.215041 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:15.215045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:15.215144 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:15.215247 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:15.215988 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.216727 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.217852 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.218244 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.218334 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:15.222325 (+ 3991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:52777: BlockingRecv error: recv got EOF from 127.4.197.131:52777 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 02:50:15 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:15 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:15.470840 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:15.461928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:37503)
0929 02:50:15.462457 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:15.462475 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:15.462595 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:15.462869 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:15.462872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:15.463287 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:15.463455 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:15.463464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.464472 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.464476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:15.465652 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:15.465667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.466541 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.466548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:15.466718 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:15.467334 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:15.467350 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:15.470589 (+ 3239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":308,"threads_started":1}
W20260929 02:50:15.471129 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:15.462010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:37503 (local address 127.4.197.190:42207)
0929 02:50:15.462209 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:15.462216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:15.462543 (+ 327us) server_negotiation.cc:407] Connection header received
0929 02:50:15.462687 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:15.462691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:15.462753 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:15.462838 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:15.463570 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.464362 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.465787 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.466366 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.466485 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:15.470854 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:37503: BlockingRecv error: recv got EOF from 127.4.197.129:37503 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 02:50:15 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:15 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:15.841614 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:15.832412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:40679)
0929 02:50:15.832946 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:15.832963 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:15.833115 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:15.833466 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:15.833471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:15.833925 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:15.834108 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:15.834119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.835187 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.835193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:15.836122 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:15.836136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.836845 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.836852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:15.837019 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:15.837609 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:15.837635 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:15.841330 (+ 3695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":297,"threads_started":1}
W20260929 02:50:15.841770 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:15.832573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:40679 (local address 127.4.197.190:42207)
0929 02:50:15.832802 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:15.832808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:15.833051 (+ 243us) server_negotiation.cc:407] Connection header received
0929 02:50:15.833248 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:15.833253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:15.833321 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:15.833408 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:15.834264 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.835050 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.836266 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:15.836692 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:15.836802 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:15.841557 (+ 4755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:40679: BlockingRecv error: recv got EOF from 127.4.197.130:40679 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 02:50:16 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:16 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:16.232699 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:16.223260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:43975)
0929 02:50:16.223702 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:16.223724 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:16.223840 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:16.224253 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:16.224258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:16.224696 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:16.224888 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:16.224899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.226003 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.226008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:16.227123 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:16.227139 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.227992 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.228000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:16.228204 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:16.228878 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:16.228897 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:16.232428 (+ 3531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
W20260929 02:50:16.232878 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:16.223360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:43975 (local address 127.4.197.190:42207)
0929 02:50:16.223528 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:16.223533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:16.223825 (+ 292us) server_negotiation.cc:407] Connection header received
0929 02:50:16.223969 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:16.223974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:16.224053 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:16.224140 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:16.225068 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.225834 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.227295 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.227821 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.227927 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:16.232623 (+ 4696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:43975: BlockingRecv error: recv got EOF from 127.4.197.131:43975 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 02:50:16 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:16 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:16.480388 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:16.471859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:42387)
0929 02:50:16.472410 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:16.472428 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:16.472550 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:16.472871 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:16.472875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:16.473277 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:16.473453 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:16.473462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.474404 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.474407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:16.475463 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:16.475476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.476197 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.476201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:16.476310 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:16.476899 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:16.476916 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:16.480128 (+ 3212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":244,"threads_started":1}
W20260929 02:50:16.480547 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:16.471887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:42387 (local address 127.4.197.190:42207)
0929 02:50:16.472076 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:16.472082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:16.472505 (+ 423us) server_negotiation.cc:407] Connection header received
0929 02:50:16.472650 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:16.472654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:16.472717 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:16.472794 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:16.473565 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.474292 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.475618 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.476053 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.476166 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:16.480355 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:42387: BlockingRecv error: recv got EOF from 127.4.197.129:42387 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 02:50:16 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:16 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:16.851078 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:16.842502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:46397)
0929 02:50:16.843117 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:16.843134 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:16.843275 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:16.843593 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:16.843598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:16.844062 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:16.844233 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:16.844242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.845239 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.845245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:16.846176 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:16.846188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.846952 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.846959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:16.847105 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:16.847643 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:16.847661 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:16.850804 (+ 3143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":263,"threads_started":1}
W20260929 02:50:16.851418 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:16.842631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:46397 (local address 127.4.197.190:42207)
0929 02:50:16.842845 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:16.842851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:16.843222 (+ 371us) server_negotiation.cc:407] Connection header received
0929 02:50:16.843405 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:16.843409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:16.843467 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:16.843546 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:16.844375 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.845099 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.846331 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:16.846782 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:16.846890 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:16.851101 (+ 4211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:46397: BlockingRecv error: recv got EOF from 127.4.197.130:46397 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 02:50:17 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:17 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:17.242144 14146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:17.233541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:51097)
0929 02:50:17.234098 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:17.234114 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:17.234262 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:17.234565 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:17.234569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:17.235021 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:17.235188 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:17.235197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.236162 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.236168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:17.237104 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:17.237117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.237798 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.237804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:17.237950 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:17.238479 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:17.238494 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:17.241902 (+ 3408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":255,"threads_started":1}
W20260929 02:50:17.242276 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:17.233633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:51097 (local address 127.4.197.190:42207)
0929 02:50:17.233818 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:17.233824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:17.234203 (+ 379us) server_negotiation.cc:407] Connection header received
0929 02:50:17.234377 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:17.234381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:17.234439 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:17.234517 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:17.235327 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.236032 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.237235 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.237658 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.237764 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:17.242069 (+ 4305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:51097: BlockingRecv error: recv got EOF from 127.4.197.131:51097 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 02:50:17 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:17 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:17.490080 14147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:17.481414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:45477)
0929 02:50:17.481964 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:17.481979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:17.482112 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:17.482443 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:17.482448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:17.482961 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:17.483134 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:17.483145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.484146 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.484151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:17.485088 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:17.485099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.485861 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.485868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:17.486008 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:17.486589 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:17.486606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:17.489748 (+ 3142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":307,"threads_started":1}
W20260929 02:50:17.490200 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:17.481551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:45477 (local address 127.4.197.190:42207)
0929 02:50:17.481741 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:17.481747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:17.482062 (+ 315us) server_negotiation.cc:407] Connection header received
0929 02:50:17.482226 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:17.482231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:17.482306 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:17.482387 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:17.483270 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.484005 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.485221 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.485666 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.485785 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:17.489967 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:45477: BlockingRecv error: recv got EOF from 127.4.197.129:45477 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 02:50:17 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:17 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:17.861387 14148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:17.852246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:41489)
0929 02:50:17.852886 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:17.852905 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:17.853062 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:17.853466 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:17.853471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:17.853996 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:17.854246 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:17.854264 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.855556 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.855561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:17.856514 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:17.856527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.857234 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.857239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:17.857359 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:17.857998 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:17.858017 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:17.861121 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":336,"threads_started":1}
W20260929 02:50:17.861539 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:17.852362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:41489 (local address 127.4.197.190:42207)
0929 02:50:17.852554 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:17.852560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:17.852994 (+ 434us) server_negotiation.cc:407] Connection header received
0929 02:50:17.853195 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:17.853200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:17.853316 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:17.853415 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:17.854442 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.855414 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.856641 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:17.857081 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:17.857175 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:17.861346 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:41489: BlockingRecv error: recv got EOF from 127.4.197.130:41489 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 02:50:18 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:18 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:18.252637 14149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:18.243022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:56197)
0929 02:50:18.243514 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:18.243531 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:18.243686 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:18.244023 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:18.244026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:18.244411 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:18.244571 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:18.244607 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.245697 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.245702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:18.246704 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:18.246718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.247681 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.247688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:18.247892 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:18.248611 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:18.248631 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:18.252370 (+ 3739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
W20260929 02:50:18.252821 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:18.243176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:56197 (local address 127.4.197.190:42207)
0929 02:50:18.243416 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:18.243422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:18.243601 (+ 179us) server_negotiation.cc:407] Connection header received
0929 02:50:18.243832 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:18.243836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:18.243902 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:18.243990 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:18.244759 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.245529 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.246874 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.247467 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.247594 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:18.252591 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:56197: BlockingRecv error: recv got EOF from 127.4.197.131:56197 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 02:50:18 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:18 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:18.499397 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:18.491142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:34595)
0929 02:50:18.491585 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:18.491601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:18.491729 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:18.491992 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:18.491995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:18.492368 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:18.492510 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:18.492519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.493596 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.493600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:18.494533 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:18.494554 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.495243 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.495247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:18.495368 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:18.495954 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:18.495971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:18.499173 (+ 3202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":226,"threads_started":1}
W20260929 02:50:18.499698 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:18.491189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:34595 (local address 127.4.197.190:42207)
0929 02:50:18.491396 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:18.491402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:18.491671 (+ 269us) server_negotiation.cc:407] Connection header received
0929 02:50:18.491830 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:18.491834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:18.491893 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:18.491965 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:18.492657 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.493429 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.494671 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.495116 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.495206 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:18.499383 (+ 4177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:34595: BlockingRecv error: recv got EOF from 127.4.197.129:34595 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 02:50:18 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:18 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:18.875147 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:18.862465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:55905)
0929 02:50:18.862947 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:18.862964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:18.863086 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:18.863381 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:18.863386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:18.863829 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:18.863972 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:18.863981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.865058 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.865061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:18.866119 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:18.866132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.866783 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.866788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:18.866944 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:18.871509 (+ 4565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:18.871529 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:18.874884 (+ 3355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":237,"threads_started":1}
W20260929 02:50:18.875309 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:18.862504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:55905 (local address 127.4.197.190:42207)
0929 02:50:18.862666 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:18.862672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:18.863040 (+ 368us) server_negotiation.cc:407] Connection header received
0929 02:50:18.863197 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:18.863201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:18.863262 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:18.863339 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:18.864142 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.864925 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.866236 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:18.866638 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:18.866735 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:18.875087 (+ 8352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:55905: BlockingRecv error: recv got EOF from 127.4.197.130:55905 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 02:50:19 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:19 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:19.262771 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:19.253874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:55757)
0929 02:50:19.254400 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:19.254418 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:19.254555 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:19.254887 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:19.254891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:19.255326 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:19.255470 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:19.255479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.256535 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.256540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:19.257650 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:19.257665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.258388 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.258393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:19.258552 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:19.259128 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:19.259152 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:19.262511 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":271,"threads_started":1}
W20260929 02:50:19.262898 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:19.253955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:55757 (local address 127.4.197.190:42207)
0929 02:50:19.254135 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:19.254140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:19.254493 (+ 353us) server_negotiation.cc:407] Connection header received
0929 02:50:19.254652 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:19.254656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:19.254716 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:19.254802 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:19.255589 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.256412 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.257788 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.258218 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.258310 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:19.262697 (+ 4387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:55757: BlockingRecv error: recv got EOF from 127.4.197.131:55757 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 02:50:19 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:19 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:19.508204 14153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:19.500270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:39673)
0929 02:50:19.500782 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:19.500797 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:19.500906 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:19.501168 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:19.501171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:19.501517 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:19.501675 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:19.501684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.502572 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.502575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:19.503580 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:19.503592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.504249 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.504253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:19.504357 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:19.504927 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:19.504946 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:19.507962 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
W20260929 02:50:19.508350 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:19.500322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:39673 (local address 127.4.197.190:42207)
0929 02:50:19.500508 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:19.500514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:19.500860 (+ 346us) server_negotiation.cc:407] Connection header received
0929 02:50:19.500986 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:19.500989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:19.501047 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:19.501130 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:19.501780 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.502478 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.503708 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.504138 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.504222 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:19.508133 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:39673: BlockingRecv error: recv got EOF from 127.4.197.129:39673 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:50:19 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:19 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:19.884485 14154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:19.876082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:37399)
0929 02:50:19.876567 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:19.876584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:19.876707 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:19.876988 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:19.876992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:19.877362 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:19.877520 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:19.877529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.878474 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.878478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:19.879516 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:19.879529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.880149 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.880153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:19.880258 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:19.880795 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:19.880811 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:19.884249 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
W20260929 02:50:19.884698 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:19.876197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:37399 (local address 127.4.197.190:42207)
0929 02:50:19.876359 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:19.876363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:19.876652 (+ 289us) server_negotiation.cc:407] Connection header received
0929 02:50:19.876809 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:19.876812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:19.876873 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:19.876956 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:19.877637 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.878352 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.879633 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:19.880029 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:19.880125 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:19.884435 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:37399: BlockingRecv error: recv got EOF from 127.4.197.130:37399 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 02:50:20 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:20 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:20.272502 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:20.263797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:43847)
0929 02:50:20.264304 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:20.264324 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:20.264456 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:20.264780 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:20.264786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:20.265241 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:20.265404 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:20.265415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.266445 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.266450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:20.267481 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:20.267494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.268231 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.268237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:20.268397 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:20.268937 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:20.268952 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:20.272273 (+ 3321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":242,"threads_started":1}
W20260929 02:50:20.272701 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:20.264000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:43847 (local address 127.4.197.190:42207)
0929 02:50:20.264291 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:20.264297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:20.264393 (+ 96us) server_negotiation.cc:407] Connection header received
0929 02:50:20.264582 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:20.264587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:20.264652 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:20.264731 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:20.265551 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.266304 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.267627 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.268075 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.268197 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:20.272454 (+ 4257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:43847: BlockingRecv error: recv got EOF from 127.4.197.131:43847 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Sep 29 02:50:20 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:20 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:20.518007 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:20.509149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:44409)
0929 02:50:20.509623 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:20.509643 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:20.509778 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:20.510064 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:20.510067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:20.510468 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:20.510635 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:20.510644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.511899 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.511905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:20.512968 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:20.512983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.513706 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.513711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:20.513837 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:20.514437 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:20.514455 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:20.517785 (+ 3330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
W20260929 02:50:20.518239 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:20.509289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:44409 (local address 127.4.197.190:42207)
0929 02:50:20.509508 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:20.509514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:20.509736 (+ 222us) server_negotiation.cc:407] Connection header received
0929 02:50:20.509874 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:20.509878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:20.509941 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:20.510022 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:20.510803 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.511742 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.513095 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.513531 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.513637 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:20.518009 (+ 4372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:44409: BlockingRecv error: recv got EOF from 127.4.197.129:44409 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 02:50:20 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:20 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:20.893785 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:20.885322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:37537)
0929 02:50:20.885830 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:20.885846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:20.885991 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:20.886309 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:20.886314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:20.886828 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:20.887062 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:20.887072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.888062 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.888068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:20.888961 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:20.888972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.889630 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.889636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:20.889815 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:20.890360 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:20.890377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:20.893473 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":302,"threads_started":1}
W20260929 02:50:20.893893 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:20.885436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:37537 (local address 127.4.197.190:42207)
0929 02:50:20.885657 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:20.885664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:20.885930 (+ 266us) server_negotiation.cc:407] Connection header received
0929 02:50:20.886119 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:20.886124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:20.886184 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:20.886263 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:20.887185 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.887945 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.889087 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:20.889489 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:20.889592 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:20.893670 (+ 4078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:37537: BlockingRecv error: recv got EOF from 127.4.197.130:37537 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 02:50:21 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:21 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:21.282176 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:21.273602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:37791)
0929 02:50:21.274084 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:21.274102 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:21.274227 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:21.274608 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:21.274613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:21.275119 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:21.275355 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:21.275369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.276367 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.276372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:21.277336 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:21.277347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.278034 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.278040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:21.278159 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:21.278733 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:21.278752 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:21.281953 (+ 3201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":245,"threads_started":1}
W20260929 02:50:21.282378 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:21.273721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:37791 (local address 127.4.197.190:42207)
0929 02:50:21.273926 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:21.273931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:21.274190 (+ 259us) server_negotiation.cc:407] Connection header received
0929 02:50:21.274350 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:21.274356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:21.274434 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:21.274514 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:21.275491 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.276232 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.277476 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.277901 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.277997 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:21.282164 (+ 4167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:37791: BlockingRecv error: recv got EOF from 127.4.197.131:37791 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 02:50:21 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:21 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:21.527383 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:21.518873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:56721)
0929 02:50:21.519362 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:21.519379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:21.519491 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:21.519779 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:21.519783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:21.520182 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:21.520344 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:21.520352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.521304 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.521307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:21.522219 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:21.522230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.522996 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.523001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:21.523126 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:21.523698 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:21.523716 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:21.527063 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":245,"threads_started":1}
W20260929 02:50:21.527525 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:21.518943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:56721 (local address 127.4.197.190:42207)
0929 02:50:21.519178 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:21.519185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:21.519451 (+ 266us) server_negotiation.cc:407] Connection header received
0929 02:50:21.519581 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:21.519585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:21.519667 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:21.519751 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:21.520469 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.521206 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.522366 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.522821 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.522939 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:21.527279 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:56721: BlockingRecv error: recv got EOF from 127.4.197.129:56721 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 02:50:21 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:21 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:21.903254 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:21.894796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:59313)
0929 02:50:21.895340 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:21.895355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:21.895484 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:21.895804 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:21.895809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:21.896197 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:21.896364 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:21.896373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.897386 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.897391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:21.898300 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:21.898311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.899030 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.899036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:21.899203 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:21.899809 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:21.899827 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:21.902905 (+ 3078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
W20260929 02:50:21.903362 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:21.894857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:59313 (local address 127.4.197.190:42207)
0929 02:50:21.895044 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:21.895051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:21.895437 (+ 386us) server_negotiation.cc:407] Connection header received
0929 02:50:21.895605 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:21.895610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:21.895682 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:21.895763 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:21.896504 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.897255 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.898431 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:21.898845 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:21.898982 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:21.903137 (+ 4155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:59313: BlockingRecv error: recv got EOF from 127.4.197.130:59313 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 02:50:22 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:22 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:22.291347 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:22.283056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:42245)
0929 02:50:22.283559 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:22.283574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:22.283688 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:22.283975 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:22.283978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:22.284362 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:22.284524 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:22.284533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.285567 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.285571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:22.286483 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:22.286494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.287294 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.287298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:22.287426 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:22.288026 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:22.288043 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:22.291096 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":259,"threads_started":1}
W20260929 02:50:22.291540 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:22.283178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:42245 (local address 127.4.197.190:42207)
0929 02:50:22.283347 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:22.283351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:22.283641 (+ 290us) server_negotiation.cc:407] Connection header received
0929 02:50:22.283798 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:22.283802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:22.283868 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:22.283950 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:22.284675 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.285439 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.286627 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.287143 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.287250 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:22.291322 (+ 4072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:42245: BlockingRecv error: recv got EOF from 127.4.197.131:42245 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 02:50:22 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:22 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:22.537628 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:22.528465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:40417)
0929 02:50:22.529000 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:22.529017 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:22.529157 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:22.529470 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:22.529475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:22.529930 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:22.530082 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:22.530092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.531123 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.531127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:22.532189 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:22.532201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.532948 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.532955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:22.533131 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:22.533765 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:22.533790 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:22.537378 (+ 3588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":310,"threads_started":1}
W20260929 02:50:22.537817 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:22.528629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:40417 (local address 127.4.197.190:42207)
0929 02:50:22.528857 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:22.528863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:22.529108 (+ 245us) server_negotiation.cc:407] Connection header received
0929 02:50:22.529276 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:22.529280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:22.529340 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:22.529423 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:22.530230 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.530971 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.532322 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.532757 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.532884 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:22.537607 (+ 4723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:40417: BlockingRecv error: recv got EOF from 127.4.197.129:40417 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 02:50:22 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:22 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:22.913204 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:22.904379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:59789)
0929 02:50:22.904910 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:22.904926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:22.905074 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:22.905490 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:22.905496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:22.905909 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:22.906058 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:22.906068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.907173 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.907177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:22.908088 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:22.908100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.908813 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.908819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:22.908990 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:22.909537 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:22.909554 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:22.912892 (+ 3338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":294,"threads_started":1}
W20260929 02:50:22.913395 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:22.904564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:59789 (local address 127.4.197.190:42207)
0929 02:50:22.904783 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:22.904789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:22.905014 (+ 225us) server_negotiation.cc:407] Connection header received
0929 02:50:22.905215 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:22.905219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:22.905284 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:22.905377 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:22.906208 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.907032 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.908229 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:22.908674 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:22.908784 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:22.913165 (+ 4381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:59789: BlockingRecv error: recv got EOF from 127.4.197.130:59789 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 02:50:23 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:23 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.131@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:23.301170 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:23.292461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.131:40863)
0929 02:50:23.292944 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:23.292961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:23.293119 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:23.293469 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:23.293475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:23.293944 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:23.294102 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:23.294113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.295219 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.295224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:23.296225 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:23.296239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.296965 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.296971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:23.297157 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:23.297720 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:23.297740 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:23.300877 (+ 3137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
W20260929 02:50:23.301319 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:23.292580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:40863 (local address 127.4.197.190:42207)
0929 02:50:23.292769 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:23.292776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:23.293052 (+ 276us) server_negotiation.cc:407] Connection header received
0929 02:50:23.293236 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:23.293243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:23.293322 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:23.293420 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:23.294255 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.295077 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.296375 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.296808 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.296937 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:23.301107 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.131:40863: BlockingRecv error: recv got EOF from 127.4.197.131:40863 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 02:50:23 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:23 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.129@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:23.548237 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:23.538880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.129:44391)
0929 02:50:23.539469 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:23.539487 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:23.539648 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:23.539965 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:23.539971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:23.540411 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:23.540620 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:23.540630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.541661 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.541665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:23.542639 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:23.542655 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.543583 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.543589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:23.543757 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:23.544370 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:23.544388 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:23.547879 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
W20260929 02:50:23.548342 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:23.539049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:44391 (local address 127.4.197.190:42207)
0929 02:50:23.539418 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:23.539425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:23.539592 (+ 167us) server_negotiation.cc:407] Connection header received
0929 02:50:23.539769 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:23.539773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:23.539835 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:23.539917 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:23.540765 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.541517 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.542806 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.543427 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.543542 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:23.548107 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.129:44391: BlockingRecv error: recv got EOF from 127.4.197.129:44391 (error 108)
Metrics: {"server-negotiator.queue_time_us":203}
Sep 29 02:50:23 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:50:23 dist-test-slave-t6c1 krb5kdc[13119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.197.130@KRBTEST.COM for oryx/127.4.197.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:50:23.925769 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:23.914459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42207 (local address 127.4.197.130:52345)
0929 02:50:23.914971 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:23.914990 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:23.915113 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:23.915446 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:23.915450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:23.915849 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:23.916019 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:23.916028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.917110 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.917115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:23.918018 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:23.918029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.918673 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.918679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:23.918848 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:23.922261 (+ 3413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:23.922277 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:23.925489 (+ 3212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.197.190:42207: Server oryx/127.4.197.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":263,"threads_started":1}
W20260929 02:50:23.925941 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:23.914580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:52345 (local address 127.4.197.190:42207)
0929 02:50:23.914768 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:23.914775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:23.915058 (+ 283us) server_negotiation.cc:407] Connection header received
0929 02:50:23.915269 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:23.915273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:23.915330 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:23.915402 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:23.916150 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.916971 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.918139 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:23.918548 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:23.918648 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:23.925746 (+ 7098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.197.130:52345: BlockingRecv error: recv got EOF from 127.4.197.130:52345 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:50:24.090976 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13706
I20260929 02:50:24.102501 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:24.091915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.4.197.129:37531)
0929 02:50:24.092440 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:24.092446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:24.092478 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:50:24.092538 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:24.092545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:24.092890 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:24.093061 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:24.094415 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:24.095472 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:24.096608 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:24.097164 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:24.097276 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:24.099518 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:24.099526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:24.099565 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:24.099607 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:24.102020 (+ 2413us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.197.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:50:24.102287 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38752: Request ticket server kudu/127.4.197.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":253,"threads_started":1}
W20260929 02:50:24.102602 14168 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38752: Request ticket server kudu/127.4.197.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:50:24.102634 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13706: Not authorized: Client connection negotiation failed: client connection to 127.4.197.129:37531: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.197.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:50:24.102815 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 13706
I20260929 02:50:24.116199 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13857
I20260929 02:50:24.127792 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:24.117053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.4.197.130:42681)
0929 02:50:24.117605 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:24.117613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:24.117729 (+ 116us) server_negotiation.cc:407] Connection header received
0929 02:50:24.117871 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:24.117877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:24.118109 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:24.118272 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:24.119828 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:24.120856 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:24.121984 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:24.122534 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:24.122626 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:24.124853 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:24.124861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:24.124875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:24.124913 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:24.127307 (+ 2394us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.197.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:50:24.127544 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60576: Request ticket server kudu/127.4.197.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":243,"threads_started":1}
W20260929 02:50:24.127892 14169 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60576: Request ticket server kudu/127.4.197.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:50:24.127877 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13857: Not authorized: Client connection negotiation failed: client connection to 127.4.197.130:42681: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.197.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:50:24.128006 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 13857
I20260929 02:50:24.140578 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13990
I20260929 02:50:24.152263 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:24.141404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.4.197.131:35709)
0929 02:50:24.141979 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:24.141988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:24.142015 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:50:24.142084 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:24.142092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:24.142458 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:24.142666 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:24.143954 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:24.145077 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:24.146206 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:24.146732 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:24.146827 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:24.149165 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:24.149172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:24.149185 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:24.149237 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:24.151752 (+ 2515us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.197.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:50:24.152003 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50586: Request ticket server kudu/127.4.197.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":262,"threads_started":1}
W20260929 02:50:24.152294 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13990: Not authorized: Client connection negotiation failed: client connection to 127.4.197.131:35709: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.197.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:50:24.152354 14170 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50586: Request ticket server kudu/127.4.197.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:50:24.152441 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 13990
I20260929 02:50:24.164968 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 13637
I20260929 02:50:24.166203 13699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:24.277537 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 13637
2026-09-29T02:50:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20764 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T02:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:50:24Z Disabled control of system clock
I20260929 02:50:24.340647 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:35411
--webserver_interface=127.4.197.190
--webserver_port=0
--builtin_ntp_servers=127.4.197.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 02:50:24.638900 14175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:24.639182 14175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:24.639263 14175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:24.646135 14175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:24.646234 14175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:24.646273 14175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:24.646327 14175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:24.646369 14175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:24.655033 14175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:35411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:24.656204 14175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:24.657831 14175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:24.668396 14180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:24.668438 14183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:24.668470 14181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:24.669456 14175 server_base.cc:1034] running on GCE node
I20260929 02:50:24.670199 14175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:24.671525 14175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:24.672744 14175 hybrid_clock.cc:648] HybridClock initialized: now 1790650224672693 us; error 46 us; skew 500 ppm
I20260929 02:50:24.673085 14175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:24.675212 14175 webserver.cc:460] Webserver started at http://127.4.197.190:37175/ using document root <none> and password file <none>
I20260929 02:50:24.675865 14175 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:24.675961 14175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:24.676249 14175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:24.678956 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26"
format_stamp: "Formatted at 2026-09-29 02:50:24 on dist-test-slave-t6c1"
server_key: "34f3f663bd9c7ce35102e6dd48d73271"
server_key_iv: "52d7c44eb0c53570534c3864cf355580"
server_key_version: "encryptionkey@0"
I20260929 02:50:24.679703 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26"
format_stamp: "Formatted at 2026-09-29 02:50:24 on dist-test-slave-t6c1"
server_key: "34f3f663bd9c7ce35102e6dd48d73271"
server_key_iv: "52d7c44eb0c53570534c3864cf355580"
server_key_version: "encryptionkey@0"
I20260929 02:50:24.685377 14175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:50:24.688555 14189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:24.689891 14175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:50:24.690056 14175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26"
format_stamp: "Formatted at 2026-09-29 02:50:24 on dist-test-slave-t6c1"
server_key: "34f3f663bd9c7ce35102e6dd48d73271"
server_key_iv: "52d7c44eb0c53570534c3864cf355580"
server_key_version: "encryptionkey@0"
I20260929 02:50:24.690246 14175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:24.709403 14175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:24.710178 14175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:24.710427 14175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:24.726441 14175 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:35411
I20260929 02:50:24.726464 14240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:35411 every 8 connection(s)
I20260929 02:50:24.727687 14175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:50:24.731238 14241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:24.735365 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 14175
I20260929 02:50:24.735525 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:50:24.736148 4886 external_mini_cluster.cc:1428] Setting key 1ed9dc4997b656c97b28ccf762fd185b
I20260929 02:50:24.741659 14241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26: Bootstrap starting.
I20260929 02:50:24.744560 14241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:24.745530 14241 log.cc:824] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:24.748617 14241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26: No bootstrap required, opened a new log
I20260929 02:50:24.749284 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:24.738061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.4.197.190:35411)
0929 02:50:24.739041 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:24.739051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:24.739095 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:50:24.740731 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:24.740752 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:24.741199 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:24.741535 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:24.742074 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:24.743405 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:24.744685 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:24.745359 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:24.745523 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:24.746660 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:24.746669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:24.746686 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:24.747361 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:24.747453 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:24.747477 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:24.747587 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:24.747800 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 02:50:24.748117 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":320,"threads_started":1}
I20260929 02:50:24.756101 14241 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35411 } }
I20260929 02:50:24.756392 14241 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:24.756464 14241 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc4f0b5f6ef4d6988e60ab30babfb26, State: Initialized, Role: FOLLOWER
I20260929 02:50:24.756950 14241 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35411 } }
I20260929 02:50:24.757138 14241 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:50:24.757230 14241 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:50:24.757345 14241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:24.759554 14241 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35411 } }
I20260929 02:50:24.759989 14241 leader_election.cc:302] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccc4f0b5f6ef4d6988e60ab30babfb26; no voters:
I20260929 02:50:24.760560 14241 leader_election.cc:288] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:50:24.760715 14246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:24.761409 14246 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [term 1 LEADER]: Becoming Leader. State: Replica: ccc4f0b5f6ef4d6988e60ab30babfb26, State: Running, Role: LEADER
I20260929 02:50:24.761952 14246 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35411 } }
I20260929 02:50:24.762583 14241 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:50:24.764473 14248 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccc4f0b5f6ef4d6988e60ab30babfb26. Latest consensus state: current_term: 1 leader_uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35411 } } }
I20260929 02:50:24.765338 14248 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:24.765035 14247 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc4f0b5f6ef4d6988e60ab30babfb26" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35411 } } }
I20260929 02:50:24.765898 14247 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:24.772305 14256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:50:24.776156 14256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:50:24.783164 14256 catalog_manager.cc:1347] Generated new cluster ID: 93ccc6c262e843cbbfc136044d8031eb
I20260929 02:50:24.783253 14256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:50:24.802884 14256 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:50:24.804033 14256 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:50:24.808475 14256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccc4f0b5f6ef4d6988e60ab30babfb26: Generated new TSK 0
I20260929 02:50:24.809162 14256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:50:24.823521 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35411
--builtin_ntp_servers=127.4.197.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 02:50:25.125221 14265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:25.125470 14265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:25.125587 14265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:25.132481 14265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:25.132575 14265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:25.132717 14265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:50:25.141301 14265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:25.142467 14265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:25.144153 14265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:25.155779 14270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:25.156003 14273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:25.156033 14271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:25.156844 14265 server_base.cc:1034] running on GCE node
I20260929 02:50:25.157392 14265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:25.158229 14265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:25.159412 14265 hybrid_clock.cc:648] HybridClock initialized: now 1790650225159371 us; error 40 us; skew 500 ppm
I20260929 02:50:25.159682 14265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:25.161650 14265 webserver.cc:460] Webserver started at http://127.4.197.129:42789/ using document root <none> and password file <none>
I20260929 02:50:25.162323 14265 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:25.162416 14265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:25.162739 14265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:25.165421 14265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "be4517881c1642fd8b13fa488a525d22"
format_stamp: "Formatted at 2026-09-29 02:50:25 on dist-test-slave-t6c1"
server_key: "dda094fa55c37ede74237766ceada00d"
server_key_iv: "aa6b2869e5fc1d2b5399dad5ad219826"
server_key_version: "encryptionkey@0"
I20260929 02:50:25.166121 14265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "be4517881c1642fd8b13fa488a525d22"
format_stamp: "Formatted at 2026-09-29 02:50:25 on dist-test-slave-t6c1"
server_key: "dda094fa55c37ede74237766ceada00d"
server_key_iv: "aa6b2869e5fc1d2b5399dad5ad219826"
server_key_version: "encryptionkey@0"
I20260929 02:50:25.170671 14265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 02:50:25.173755 14279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:25.174995 14265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:50:25.175132 14265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "be4517881c1642fd8b13fa488a525d22"
format_stamp: "Formatted at 2026-09-29 02:50:25 on dist-test-slave-t6c1"
server_key: "dda094fa55c37ede74237766ceada00d"
server_key_iv: "aa6b2869e5fc1d2b5399dad5ad219826"
server_key_version: "encryptionkey@0"
I20260929 02:50:25.175321 14265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:25.194005 14265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:25.194726 14265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:25.195010 14265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:25.195845 14265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:25.197094 14265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:25.197194 14265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:25.197299 14265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:25.197366 14265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:25.225085 14265 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:46443
I20260929 02:50:25.225143 14391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:46443 every 8 connection(s)
I20260929 02:50:25.226248 14265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:50:25.231817 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 14265
I20260929 02:50:25.232010 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:50:25.232617 4886 external_mini_cluster.cc:1428] Setting key f78abed07fe954f45e095d4ce4878a27
I20260929 02:50:25.235271 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35411
--builtin_ntp_servers=127.4.197.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 02:50:25.237931 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:25.228550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:38857 (local address 127.4.197.190:35411)
0929 02:50:25.228708 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:25.228712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:25.229967 (+ 1255us) server_negotiation.cc:407] Connection header received
0929 02:50:25.231667 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:25.231673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:25.231754 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:25.231867 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:25.232859 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:25.233685 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:25.235465 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:25.235849 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:25.235915 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:25.237249 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:25.237254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:25.237259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:25.237279 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:25.237314 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:25.237327 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:25.237416 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:25.237573 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 02:50:25.237654 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:50:25.238844 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:25.229012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:35411 (local address 127.4.197.129:38857)
0929 02:50:25.229824 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:25.229857 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:25.231423 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:25.232064 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:25.232074 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:25.232180 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:25.232709 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:25.232727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:25.233844 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:25.233848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:25.235307 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:25.235319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:25.236148 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:25.236152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:25.236334 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:25.236960 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:50:25.236962 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:25.236996 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:50:25.237066 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:25.237423 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:25.237428 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:25.237433 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:50:25.237550 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 02:50:25.237811 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":239,"threads_started":1}
I20260929 02:50:25.240269 14392 heartbeater.cc:347] Connected to a master server at 127.4.197.190:35411
I20260929 02:50:25.240545 14392 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:25.241357 14392 heartbeater.cc:511] Master 127.4.197.190:35411 requested a full tablet report, sending...
I20260929 02:50:25.243350 14206 ts_manager.cc:194] Registered new tserver with Master: be4517881c1642fd8b13fa488a525d22 (127.4.197.129:46443)
I20260929 02:50:25.245219 14206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.129:38857
W20260929 02:50:25.537472 14395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:25.537732 14395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:25.537809 14395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:25.544579 14395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:25.544659 14395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:25.544760 14395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:50:25.553146 14395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:25.554227 14395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:25.555886 14395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:25.567961 14401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:25.567957 14400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:25.568305 14403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:25.568560 14395 server_base.cc:1034] running on GCE node
I20260929 02:50:25.569144 14395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:25.569984 14395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:25.571177 14395 hybrid_clock.cc:648] HybridClock initialized: now 1790650225571132 us; error 35 us; skew 500 ppm
I20260929 02:50:25.571460 14395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:25.573482 14395 webserver.cc:460] Webserver started at http://127.4.197.130:42475/ using document root <none> and password file <none>
I20260929 02:50:25.574084 14395 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:25.574185 14395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:25.574462 14395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:25.577069 14395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "ce8561ba7837483f88835dd2eeaabf7d"
format_stamp: "Formatted at 2026-09-29 02:50:25 on dist-test-slave-t6c1"
server_key: "1d2bb56cde42bbea0d20a69793514b42"
server_key_iv: "cf15af576788669517a7f3986b3fff3a"
server_key_version: "encryptionkey@0"
I20260929 02:50:25.577770 14395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "ce8561ba7837483f88835dd2eeaabf7d"
format_stamp: "Formatted at 2026-09-29 02:50:25 on dist-test-slave-t6c1"
server_key: "1d2bb56cde42bbea0d20a69793514b42"
server_key_iv: "cf15af576788669517a7f3986b3fff3a"
server_key_version: "encryptionkey@0"
I20260929 02:50:25.582615 14395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:50:25.585963 14409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:25.587245 14395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:50:25.587386 14395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "ce8561ba7837483f88835dd2eeaabf7d"
format_stamp: "Formatted at 2026-09-29 02:50:25 on dist-test-slave-t6c1"
server_key: "1d2bb56cde42bbea0d20a69793514b42"
server_key_iv: "cf15af576788669517a7f3986b3fff3a"
server_key_version: "encryptionkey@0"
I20260929 02:50:25.587565 14395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:25.603936 14395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:25.604686 14395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:25.604940 14395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:25.605909 14395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:25.607240 14395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:25.607331 14395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:25.607432 14395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:25.607501 14395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:25.635660 14395 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:40043
I20260929 02:50:25.635727 14521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:40043 every 8 connection(s)
I20260929 02:50:25.636829 14395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:50:25.646682 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 14395
I20260929 02:50:25.646826 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:50:25.647372 4886 external_mini_cluster.cc:1428] Setting key 37019f46f46891c0270a8cbdb97b6168
I20260929 02:50:25.648249 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:25.639210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:43191 (local address 127.4.197.190:35411)
0929 02:50:25.639376 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:25.639381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:25.640750 (+ 1369us) server_negotiation.cc:407] Connection header received
0929 02:50:25.642418 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:25.642423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:25.642498 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:25.642583 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:25.643601 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:25.644294 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:25.645764 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:25.646145 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:25.646251 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:25.647501 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:25.647507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:25.647511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:25.647528 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:25.647557 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:25.647568 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:25.647639 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:25.647824 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 02:50:25.647913 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:50:25.649168 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:25.639753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:35411 (local address 127.4.197.130:43191)
0929 02:50:25.640606 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:25.640640 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:25.642158 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:25.642806 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:25.642814 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:25.642895 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:25.643441 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:25.643457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:25.644435 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:25.644439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:25.645633 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:25.645648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:25.646338 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:25.646345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:25.646558 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:25.647238 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:50:25.647241 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:25.647275 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:50:25.647339 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:25.647647 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:25.647657 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:25.647664 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:50:25.647802 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 02:50:25.648143 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":276,"threads_started":1}
I20260929 02:50:25.650075 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35411
--builtin_ntp_servers=127.4.197.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 02:50:25.650357 14522 heartbeater.cc:347] Connected to a master server at 127.4.197.190:35411
I20260929 02:50:25.650599 14522 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:25.651264 14522 heartbeater.cc:511] Master 127.4.197.190:35411 requested a full tablet report, sending...
I20260929 02:50:25.652556 14206 ts_manager.cc:194] Registered new tserver with Master: ce8561ba7837483f88835dd2eeaabf7d (127.4.197.130:40043)
I20260929 02:50:25.653580 14206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.130:43191
W20260929 02:50:25.962877 14525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:25.963171 14525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:25.963353 14525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:25.970695 14525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:25.970793 14525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:25.970978 14525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:50:25.980121 14525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:25.981355 14525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:25.983218 14525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:25.996197 14530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:25.996152 14533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:25.996241 14531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:25.997036 14525 server_base.cc:1034] running on GCE node
I20260929 02:50:25.997637 14525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:25.998570 14525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:25.999781 14525 hybrid_clock.cc:648] HybridClock initialized: now 1790650225999724 us; error 44 us; skew 500 ppm
I20260929 02:50:26.000157 14525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:26.002326 14525 webserver.cc:460] Webserver started at http://127.4.197.131:35735/ using document root <none> and password file <none>
I20260929 02:50:26.003031 14525 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:26.003144 14525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:26.003438 14525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:26.006217 14525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "3dd2e94abe0f49f7978d377b52b6bf91"
format_stamp: "Formatted at 2026-09-29 02:50:26 on dist-test-slave-t6c1"
server_key: "9f09ee2840c9555906846af3bb83bc9f"
server_key_iv: "5cacb105cddfd7e80f0c070f638c1da3"
server_key_version: "encryptionkey@0"
I20260929 02:50:26.006990 14525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "3dd2e94abe0f49f7978d377b52b6bf91"
format_stamp: "Formatted at 2026-09-29 02:50:26 on dist-test-slave-t6c1"
server_key: "9f09ee2840c9555906846af3bb83bc9f"
server_key_iv: "5cacb105cddfd7e80f0c070f638c1da3"
server_key_version: "encryptionkey@0"
I20260929 02:50:26.011832 14525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:50:26.015051 14539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:26.016327 14525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:50:26.016472 14525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "3dd2e94abe0f49f7978d377b52b6bf91"
format_stamp: "Formatted at 2026-09-29 02:50:26 on dist-test-slave-t6c1"
server_key: "9f09ee2840c9555906846af3bb83bc9f"
server_key_iv: "5cacb105cddfd7e80f0c070f638c1da3"
server_key_version: "encryptionkey@0"
I20260929 02:50:26.016657 14525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:26.027138 14525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:26.027902 14525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:26.028177 14525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:26.029031 14525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:26.030405 14525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:26.030495 14525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:26.030597 14525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:26.030669 14525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:26.059103 14525 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:46165
I20260929 02:50:26.059157 14651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:46165 every 8 connection(s)
I20260929 02:50:26.060346 14525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:50:26.068858 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 14525
I20260929 02:50:26.068984 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:50:26.069401 4886 external_mini_cluster.cc:1428] Setting key b523c4026ae37f732cae40d991a996b5
I20260929 02:50:26.072033 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:26.062695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:34921 (local address 127.4.197.190:35411)
0929 02:50:26.062863 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:26.062868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:26.064442 (+ 1574us) server_negotiation.cc:407] Connection header received
0929 02:50:26.066106 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:26.066110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:26.066186 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:26.066274 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:26.067276 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:26.067949 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:26.069438 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:26.069997 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:26.070109 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:26.071332 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:26.071338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:26.071342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:26.071357 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:26.071386 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:26.071397 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:26.071466 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:26.071663 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 02:50:26.071750 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:50:26.072960 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:26.063281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:35411 (local address 127.4.197.131:34921)
0929 02:50:26.064298 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:26.064341 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:26.065877 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:26.066462 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:26.066470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:26.066560 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:26.067134 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:26.067153 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:26.068057 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:26.068060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:26.069307 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:26.069318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:26.070211 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:26.070217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:26.070426 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:26.071053 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:50:26.071056 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:26.071092 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:50:26.071165 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:26.071475 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:26.071485 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:26.071493 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:50:26.071635 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 02:50:26.071993 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":301,"threads_started":1}
I20260929 02:50:26.074077 14652 heartbeater.cc:347] Connected to a master server at 127.4.197.190:35411
I20260929 02:50:26.074342 14652 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:26.075029 14652 heartbeater.cc:511] Master 127.4.197.190:35411 requested a full tablet report, sending...
I20260929 02:50:26.076256 14206 ts_manager.cc:194] Registered new tserver with Master: 3dd2e94abe0f49f7978d377b52b6bf91 (127.4.197.131:46165)
I20260929 02:50:26.076993 14206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.131:34921
I20260929 02:50:26.084005 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:50:26.089461 14244 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:26.088165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.4.197.190:35411)
0929 02:50:26.088296 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:26.088299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:26.088716 (+ 417us) server_negotiation.cc:407] Connection header received
0929 02:50:26.088855 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:26.088858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:26.088916 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:26.088987 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:26.089252 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46302: BlockingRecv error: recv got EOF from 127.0.0.1:46302 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 02:50:26.090346 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 14265
I20260929 02:50:26.097059 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:26.091149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.4.197.129:46443)
0929 02:50:26.091661 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:26.091677 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:26.091703 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:50:26.091764 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:26.091770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:26.091939 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:26.092051 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:26.092334 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:26.093260 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:26.094422 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:26.094979 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:26.095070 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:26.095880 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:26.095887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:26.095897 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:26.096392 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:26.096432 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:26.096454 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:26.096534 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:26.096696 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 02:50:26.096804 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":278,"threads_started":1}
I20260929 02:50:26.098129 14387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:26.176285 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 14265
I20260929 02:50:26.188604 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 14395
I20260929 02:50:26.196610 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:26.189403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.4.197.130:40043)
0929 02:50:26.189969 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:26.189994 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:26.190034 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:50:26.190117 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:26.190125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:26.190428 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:26.190659 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:26.191047 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:26.192144 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:26.193629 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:26.194245 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:26.194344 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:26.195305 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:26.195314 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:26.195326 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:26.195913 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:26.195957 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:26.195977 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:26.196068 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:26.196145 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 02:50:26.196258 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
I20260929 02:50:26.197731 14517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:26.275986 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 14395
I20260929 02:50:26.288049 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 14525
I20260929 02:50:26.295477 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:26.288888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.4.197.131:46165)
0929 02:50:26.289486 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:26.289493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:26.289524 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:50:26.289581 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:26.289587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:26.289807 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:26.289929 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:26.290328 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:26.291425 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:26.292551 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:26.293099 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:26.293195 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:26.294121 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:26.294129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:26.294142 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:26.294703 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:26.294748 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:26.294766 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:26.294854 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:26.295008 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 02:50:26.295118 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
I20260929 02:50:26.296634 14647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:26.375157 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 14525
I20260929 02:50:26.387084 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 14175
I20260929 02:50:26.388175 14236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:26.486169 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 14175
2026-09-29T02:50:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2204 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T02:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:50:26Z Disabled control of system clock
I20260929 02:50:26.546842 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36305
--webserver_interface=127.4.197.190
--webserver_port=0
--builtin_ntp_servers=127.4.197.148:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:36305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:50:26.844477 14671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:26.844770 14671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:26.844890 14671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:26.851749 14671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:26.851835 14671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:26.851893 14671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:26.851967 14671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:26.852023 14671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:26.860524 14671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:36305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:26.861711 14671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:26.863375 14671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:26.874117 14679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:26.874070 14677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:26.874449 14676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:26.875026 14671 server_base.cc:1034] running on GCE node
I20260929 02:50:26.875736 14671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:26.877008 14671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:26.878201 14671 hybrid_clock.cc:648] HybridClock initialized: now 1790650226878159 us; error 33 us; skew 500 ppm
I20260929 02:50:26.878499 14671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:26.880520 14671 webserver.cc:460] Webserver started at http://127.4.197.190:43981/ using document root <none> and password file <none>
I20260929 02:50:26.881134 14671 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:26.881229 14671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:26.881515 14671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:26.884189 14671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "b4709532aef348558f7e43b014890862"
format_stamp: "Formatted at 2026-09-29 02:50:26 on dist-test-slave-t6c1"
server_key: "4be9be7f828256ea100747a14540dc38"
server_key_iv: "c30380f9d68b24bc2eafb8bfd239cd9d"
server_key_version: "encryptionkey@0"
I20260929 02:50:26.884902 14671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "b4709532aef348558f7e43b014890862"
format_stamp: "Formatted at 2026-09-29 02:50:26 on dist-test-slave-t6c1"
server_key: "4be9be7f828256ea100747a14540dc38"
server_key_iv: "c30380f9d68b24bc2eafb8bfd239cd9d"
server_key_version: "encryptionkey@0"
I20260929 02:50:26.889483 14671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:50:26.892612 14685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:26.893795 14671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:50:26.893934 14671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "b4709532aef348558f7e43b014890862"
format_stamp: "Formatted at 2026-09-29 02:50:26 on dist-test-slave-t6c1"
server_key: "4be9be7f828256ea100747a14540dc38"
server_key_iv: "c30380f9d68b24bc2eafb8bfd239cd9d"
server_key_version: "encryptionkey@0"
I20260929 02:50:26.894109 14671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:26.899250 14671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:26.899911 14671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:26.900162 14671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:26.915935 14671 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:36305
I20260929 02:50:26.915970 14736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:36305 every 8 connection(s)
I20260929 02:50:26.917055 14671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:50:26.920360 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:26.922446 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 14671
I20260929 02:50:26.922626 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:50:26.923230 4886 external_mini_cluster.cc:1428] Setting key 61c39455a8a87cc03a2d6d8b6f6af612
I20260929 02:50:26.932056 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:26.925007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.4.197.190:36305)
0929 02:50:26.926013 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:26.926027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:26.926081 (+ 54us) server_negotiation.cc:407] Connection header received
0929 02:50:26.928326 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:26.928359 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:26.928872 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:26.929282 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:26.930230 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:26.930237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:26.930253 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:26.930269 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:26.930344 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:26.930365 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:26.930577 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:26.930711 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:50:26.930969 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":302,"threads_started":1}
I20260929 02:50:26.932660 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862: Bootstrap starting.
I20260929 02:50:26.935467 14737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:26.936527 14737 log.cc:824] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:26.938819 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862: No bootstrap required, opened a new log
I20260929 02:50:26.945603 14737 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4709532aef348558f7e43b014890862" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36305 } }
I20260929 02:50:26.945899 14737 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:26.945981 14737 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4709532aef348558f7e43b014890862, State: Initialized, Role: FOLLOWER
I20260929 02:50:26.946449 14737 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4709532aef348558f7e43b014890862" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36305 } }
I20260929 02:50:26.946662 14737 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:50:26.946775 14737 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:50:26.946954 14737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:26.948954 14737 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4709532aef348558f7e43b014890862" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36305 } }
I20260929 02:50:26.949381 14737 leader_election.cc:302] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4709532aef348558f7e43b014890862; no voters:
I20260929 02:50:26.949903 14737 leader_election.cc:288] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:50:26.949985 14742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:26.950963 14742 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [term 1 LEADER]: Becoming Leader. State: Replica: b4709532aef348558f7e43b014890862, State: Running, Role: LEADER
I20260929 02:50:26.951435 14742 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4709532aef348558f7e43b014890862" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36305 } }
I20260929 02:50:26.951766 14737 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:50:26.953852 14744 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4709532aef348558f7e43b014890862. Latest consensus state: current_term: 1 leader_uuid: "b4709532aef348558f7e43b014890862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4709532aef348558f7e43b014890862" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36305 } } }
I20260929 02:50:26.953929 14743 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4709532aef348558f7e43b014890862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4709532aef348558f7e43b014890862" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36305 } } }
I20260929 02:50:26.954308 14744 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:26.954316 14743 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:26.955871 14749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:50:26.960546 14749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:50:26.968707 14749 catalog_manager.cc:1347] Generated new cluster ID: cff2900a534e437e94788e04ce8979e5
I20260929 02:50:26.968796 14749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:50:26.977635 14749 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:50:26.978014 14749 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:50:26.985085 14749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4709532aef348558f7e43b014890862: Generated new TSK 0
I20260929 02:50:26.985821 14749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:50:26.991689 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36305
--builtin_ntp_servers=127.4.197.148:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:50:27.307255 14761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:27.307484 14761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:27.307582 14761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:27.314453 14761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:27.314545 14761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:27.314646 14761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:50:27.323364 14761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:27.324416 14761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:27.326043 14761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:27.337677 14766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:27.337652 14767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:27.338167 14769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:27.338469 14761 server_base.cc:1034] running on GCE node
I20260929 02:50:27.339041 14761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:27.339850 14761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:27.341039 14761 hybrid_clock.cc:648] HybridClock initialized: now 1790650227340990 us; error 43 us; skew 500 ppm
I20260929 02:50:27.341315 14761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:27.343318 14761 webserver.cc:460] Webserver started at http://127.4.197.129:39927/ using document root <none> and password file <none>
I20260929 02:50:27.343904 14761 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:27.343992 14761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:27.344277 14761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:27.346771 14761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "014d51ab89b6484cb11ea3c4d932d6cf"
format_stamp: "Formatted at 2026-09-29 02:50:27 on dist-test-slave-t6c1"
server_key: "606914fa428629b6129b48591ede10b2"
server_key_iv: "3931f243b4498775c868b9d81ea732c2"
server_key_version: "encryptionkey@0"
I20260929 02:50:27.347530 14761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "014d51ab89b6484cb11ea3c4d932d6cf"
format_stamp: "Formatted at 2026-09-29 02:50:27 on dist-test-slave-t6c1"
server_key: "606914fa428629b6129b48591ede10b2"
server_key_iv: "3931f243b4498775c868b9d81ea732c2"
server_key_version: "encryptionkey@0"
I20260929 02:50:27.352061 14761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:50:27.355167 14775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:27.356335 14761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:50:27.356470 14761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "014d51ab89b6484cb11ea3c4d932d6cf"
format_stamp: "Formatted at 2026-09-29 02:50:27 on dist-test-slave-t6c1"
server_key: "606914fa428629b6129b48591ede10b2"
server_key_iv: "3931f243b4498775c868b9d81ea732c2"
server_key_version: "encryptionkey@0"
I20260929 02:50:27.356670 14761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:27.361654 14761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:27.362313 14761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:27.362509 14761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:27.363324 14761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:27.364529 14761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:27.364609 14761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:27.364691 14761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:27.364742 14761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:27.393114 14761 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:44953
I20260929 02:50:27.393177 14887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:44953 every 8 connection(s)
I20260929 02:50:27.394277 14761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:50:27.401760 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:27.396632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:42669 (local address 127.4.197.190:36305)
0929 02:50:27.396812 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:27.396819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:27.398112 (+ 1293us) server_negotiation.cc:407] Connection header received
0929 02:50:27.399796 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:27.399802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:27.399863 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:27.399946 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:27.401105 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:27.401111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:27.401116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:27.401134 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:27.401165 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:27.401176 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:27.401273 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:27.401440 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 02:50:27.401500 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:50:27.402593 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:27.397064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36305 (local address 127.4.197.129:42669)
0929 02:50:27.397977 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:27.398022 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:27.399551 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:27.400186 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:27.400194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:27.400283 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:27.400829 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:50:27.400831 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:27.400866 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:50:27.400943 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:27.401256 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:27.401265 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:27.401272 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:50:27.401504 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 02:50:27.401737 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":236,"threads_started":1}
I20260929 02:50:27.403213 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 14761
I20260929 02:50:27.403405 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:50:27.403975 14888 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36305
I20260929 02:50:27.404147 4886 external_mini_cluster.cc:1428] Setting key 4a433ed068ac039c38b1627334f43a98
I20260929 02:50:27.404232 14888 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:27.404749 14888 heartbeater.cc:511] Master 127.4.197.190:36305 requested a full tablet report, sending...
I20260929 02:50:27.406553 14702 ts_manager.cc:194] Registered new tserver with Master: 014d51ab89b6484cb11ea3c4d932d6cf (127.4.197.129:44953)
I20260929 02:50:27.406664 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36305
--builtin_ntp_servers=127.4.197.148:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:50:27.711741 14891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:27.711997 14891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:27.712119 14891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:27.719489 14891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:27.719584 14891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:27.719732 14891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:50:27.728639 14891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:27.730173 14891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:27.732671 14891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:27.747059 14896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:27.747035 14897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:27.746961 14899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:27.748067 14891 server_base.cc:1034] running on GCE node
I20260929 02:50:27.748703 14891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:27.749636 14891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:27.750834 14891 hybrid_clock.cc:648] HybridClock initialized: now 1790650227750790 us; error 37 us; skew 500 ppm
I20260929 02:50:27.751171 14891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:27.753484 14891 webserver.cc:460] Webserver started at http://127.4.197.130:41355/ using document root <none> and password file <none>
I20260929 02:50:27.754204 14891 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:27.754294 14891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:27.754621 14891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:27.759037 14891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "5df7b3c9a31c4d19b63ed0e923c65656"
format_stamp: "Formatted at 2026-09-29 02:50:27 on dist-test-slave-t6c1"
server_key: "27fc403ca70d458c8c8a6a5d6b7017a6"
server_key_iv: "30184a9fb7173ddc4602b89135f785aa"
server_key_version: "encryptionkey@0"
I20260929 02:50:27.759861 14891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "5df7b3c9a31c4d19b63ed0e923c65656"
format_stamp: "Formatted at 2026-09-29 02:50:27 on dist-test-slave-t6c1"
server_key: "27fc403ca70d458c8c8a6a5d6b7017a6"
server_key_iv: "30184a9fb7173ddc4602b89135f785aa"
server_key_version: "encryptionkey@0"
I20260929 02:50:27.765100 14891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 02:50:27.768507 14905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:27.769841 14891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:50:27.769979 14891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "5df7b3c9a31c4d19b63ed0e923c65656"
format_stamp: "Formatted at 2026-09-29 02:50:27 on dist-test-slave-t6c1"
server_key: "27fc403ca70d458c8c8a6a5d6b7017a6"
server_key_iv: "30184a9fb7173ddc4602b89135f785aa"
server_key_version: "encryptionkey@0"
I20260929 02:50:27.770169 14891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:27.775552 14891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:27.776261 14891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:27.776496 14891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:27.777303 14891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:27.778611 14891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:27.778707 14891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:27.778805 14891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:27.778872 14891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:27.807775 14891 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:33241
I20260929 02:50:27.807828 15017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:33241 every 8 connection(s)
I20260929 02:50:27.809039 14891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:50:27.814477 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 14891
I20260929 02:50:27.814651 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:50:27.815099 4886 external_mini_cluster.cc:1428] Setting key 0dd66a168d276fa6a6a04077415a3d8c
I20260929 02:50:27.816819 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:27.811384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:45565 (local address 127.4.197.190:36305)
0929 02:50:27.811554 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:27.811558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:27.812903 (+ 1345us) server_negotiation.cc:407] Connection header received
0929 02:50:27.814535 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:27.814541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:27.814608 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:27.814707 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:27.816087 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:27.816093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:27.816098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:27.816118 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:27.816149 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:27.816160 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:27.816261 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:27.816450 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 02:50:27.816515 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:50:27.817833 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:27.811886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36305 (local address 127.4.197.130:45565)
0929 02:50:27.812747 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:27.812782 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:27.814263 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:27.814967 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:27.814978 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:27.815082 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:27.815776 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:50:27.815779 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:27.815834 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:50:27.815929 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:27.816238 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:27.816248 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:27.816256 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:50:27.816533 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 02:50:27.816821 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":240,"threads_started":1}
I20260929 02:50:27.817889 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36305
--builtin_ntp_servers=127.4.197.148:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 02:50:27.819205 15018 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36305
I20260929 02:50:27.819528 15018 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:27.820226 15018 heartbeater.cc:511] Master 127.4.197.190:36305 requested a full tablet report, sending...
I20260929 02:50:27.821766 14702 ts_manager.cc:194] Registered new tserver with Master: 5df7b3c9a31c4d19b63ed0e923c65656 (127.4.197.130:33241)
W20260929 02:50:28.134151 15021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:28.134454 15021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:28.134577 15021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:28.141852 15021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:28.141968 15021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:28.142104 15021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:50:28.151322 15021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:28.152654 15021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:28.154440 15021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:28.167470 15026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:28.167508 15027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:28.167562 15029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:28.168756 15021 server_base.cc:1034] running on GCE node
I20260929 02:50:28.169442 15021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:28.170465 15021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:28.171669 15021 hybrid_clock.cc:648] HybridClock initialized: now 1790650228171609 us; error 52 us; skew 500 ppm
I20260929 02:50:28.171999 15021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:28.174355 15021 webserver.cc:460] Webserver started at http://127.4.197.131:45995/ using document root <none> and password file <none>
I20260929 02:50:28.175103 15021 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:28.175203 15021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:28.175498 15021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:28.178236 15021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "2bed290fab1a44d5a0011c05be8cdaa7"
format_stamp: "Formatted at 2026-09-29 02:50:28 on dist-test-slave-t6c1"
server_key: "9f2371ee831a9cbe03503c3ba9c0657b"
server_key_iv: "d554a9d2fb83fa7a1972a6aa2254d903"
server_key_version: "encryptionkey@0"
I20260929 02:50:28.179009 15021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "2bed290fab1a44d5a0011c05be8cdaa7"
format_stamp: "Formatted at 2026-09-29 02:50:28 on dist-test-slave-t6c1"
server_key: "9f2371ee831a9cbe03503c3ba9c0657b"
server_key_iv: "d554a9d2fb83fa7a1972a6aa2254d903"
server_key_version: "encryptionkey@0"
I20260929 02:50:28.183951 15021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:50:28.187335 15035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:28.188607 15021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:50:28.188753 15021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "2bed290fab1a44d5a0011c05be8cdaa7"
format_stamp: "Formatted at 2026-09-29 02:50:28 on dist-test-slave-t6c1"
server_key: "9f2371ee831a9cbe03503c3ba9c0657b"
server_key_iv: "d554a9d2fb83fa7a1972a6aa2254d903"
server_key_version: "encryptionkey@0"
I20260929 02:50:28.188943 15021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:28.194190 15021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:28.194908 15021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:28.195199 15021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:28.196045 15021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:28.197351 15021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:28.197439 15021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:28.197543 15021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:28.197609 15021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:28.224998 15021 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:46161
I20260929 02:50:28.225049 15147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:46161 every 8 connection(s)
I20260929 02:50:28.226244 15021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:50:28.230283 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 15021
I20260929 02:50:28.230464 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:50:28.230886 4886 external_mini_cluster.cc:1428] Setting key b5095bc4a930b694297a161183ea4f51
I20260929 02:50:28.234316 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:28.228686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:42363 (local address 127.4.197.190:36305)
0929 02:50:28.228904 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:28.228908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:28.230313 (+ 1405us) server_negotiation.cc:407] Connection header received
0929 02:50:28.232207 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:28.232212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:28.232286 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:28.232381 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:28.233721 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:28.233727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:28.233732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:28.233749 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:28.233782 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:28.233794 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:28.233896 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:28.234070 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 02:50:28.234122 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:50:28.235410 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:28.229186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36305 (local address 127.4.197.131:42363)
0929 02:50:28.230126 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:28.230172 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:28.231916 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:28.232627 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:28.232638 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:28.232778 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:28.233419 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:50:28.233422 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:28.233462 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:50:28.233563 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:28.233878 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:28.233886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:28.233892 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:50:28.234121 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 02:50:28.234349 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":269,"threads_started":1}
I20260929 02:50:28.236807 15148 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36305
I20260929 02:50:28.237107 15148 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:28.237725 15148 heartbeater.cc:511] Master 127.4.197.190:36305 requested a full tablet report, sending...
I20260929 02:50:28.239095 14702 ts_manager.cc:194] Registered new tserver with Master: 2bed290fab1a44d5a0011c05be8cdaa7 (127.4.197.131:46161)
I20260929 02:50:28.246315 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:50:28.252470 14740 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:50:28.250755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.4.197.190:36305)
0929 02:50:28.250995 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:28.251001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:28.251468 (+ 467us) server_negotiation.cc:407] Connection header received
0929 02:50:28.251663 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:28.251669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:28.251725 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:28.251806 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:28.252175 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50330: BlockingRecv error: recv got EOF from 127.0.0.1:50330 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:50:28.252897 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 14761
I20260929 02:50:28.256484 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:28.253899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.4.197.129:44953)
0929 02:50:28.254369 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:28.254376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:28.254401 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:50:28.254455 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:28.254461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:28.254676 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:28.254798 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:28.255755 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:28.255761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:28.255772 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:28.255789 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:28.255827 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:28.255841 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:28.255957 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:28.256091 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:50:28.256191 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260929 02:50:28.257702 14883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:28.337152 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 14761
I20260929 02:50:28.350044 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 14891
I20260929 02:50:28.353561 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:28.351017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.4.197.130:33241)
0929 02:50:28.351529 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:28.351537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:28.351564 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:50:28.351637 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:28.351646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:28.351869 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:28.352000 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:28.352936 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:28.352943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:28.352955 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:28.352973 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:28.353015 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:28.353039 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:28.353142 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:28.353193 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 02:50:28.353276 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":233,"threads_started":1}
I20260929 02:50:28.354784 15013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:28.433578 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 14891
I20260929 02:50:28.446022 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 15021
I20260929 02:50:28.449842 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:28.447014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.4.197.131:46161)
0929 02:50:28.447702 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:28.447709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:28.447741 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:50:28.447807 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:28.447814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:28.448065 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:28.448193 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:28.449187 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:28.449193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:28.449206 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:28.449221 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:28.449258 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:28.449271 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:28.449395 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:28.449442 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 02:50:28.449523 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":363,"threads_started":1}
I20260929 02:50:28.451157 15143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:28.531386 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 15021
I20260929 02:50:28.544231 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 14671
I20260929 02:50:28.545529 14732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:28.644322 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 14671
2026-09-29T02:50:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2157 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:50:28 dist-test-slave-t6c1 krb5kdc[15166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:50:28 dist-test-slave-t6c1 krb5kdc[15166](info): set up 2 sockets
Sep 29 02:50:28 dist-test-slave-t6c1 krb5kdc[15166](info): commencing operation
krb5kdc: starting...
W20260929 02:50:29.655751 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:50:29 dist-test-slave-t6c1 krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:50:29.874468 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45975
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:30.196514 15179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:30.196822 15179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:30.196954 15179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:30.204128 15179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:30.204238 15179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:30.204311 15179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:30.204385 15179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:30.204442 15179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:30.213919 15179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:45975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:30.215195 15179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:30.216404 15179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:30.227152 15187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:30.227209 15185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:30.227391 15184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:30.228209 15179 server_base.cc:1034] running on GCE node
I20260929 02:50:30.228921 15179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:30.230268 15179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:30.231453 15179 hybrid_clock.cc:648] HybridClock initialized: now 1790650230231407 us; error 37 us; skew 500 ppm
I20260929 02:50:30.231786 15179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:30 dist-test-slave-t6c1 krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:30.234711 15179 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:50:30.236167 15179 webserver.cc:460] Webserver started at http://127.4.197.190:46459/ using document root <none> and password file <none>
I20260929 02:50:30.236814 15179 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:30.236927 15179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:30.237205 15179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:30.240150 15179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "54f2bab8b9b2417da3cc4dc8dc562656"
format_stamp: "Formatted at 2026-09-29 02:50:30 on dist-test-slave-t6c1"
server_key: "ad93afeb51e21176fce75dbb6008bb58"
server_key_iv: "fd5736e2c26baad1a5b17f18ae204579"
server_key_version: "encryptionkey@0"
I20260929 02:50:30.240898 15179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "54f2bab8b9b2417da3cc4dc8dc562656"
format_stamp: "Formatted at 2026-09-29 02:50:30 on dist-test-slave-t6c1"
server_key: "ad93afeb51e21176fce75dbb6008bb58"
server_key_iv: "fd5736e2c26baad1a5b17f18ae204579"
server_key_version: "encryptionkey@0"
I20260929 02:50:30.246189 15179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:50:30.249804 15194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:30.251246 15179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:50:30.251410 15179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "54f2bab8b9b2417da3cc4dc8dc562656"
format_stamp: "Formatted at 2026-09-29 02:50:30 on dist-test-slave-t6c1"
server_key: "ad93afeb51e21176fce75dbb6008bb58"
server_key_iv: "fd5736e2c26baad1a5b17f18ae204579"
server_key_version: "encryptionkey@0"
I20260929 02:50:30.251601 15179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:30.263782 15179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:30.267383 15179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:30.267642 15179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:30.284091 15179 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:45975
I20260929 02:50:30.284122 15245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:45975 every 8 connection(s)
I20260929 02:50:30.285456 15179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:50:30.289081 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:30.294824 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 15179
I20260929 02:50:30.295060 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:50:30.295655 4886 external_mini_cluster.cc:1428] Setting key 87b985c17bc83b5cd6cd77914a229172
I20260929 02:50:30.299449 15246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656: Bootstrap starting.
I20260929 02:50:30.302910 15246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:30.303907 15246 log.cc:824] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:30.306344 15246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656: No bootstrap required, opened a new log
Sep 29 02:50:30 dist-test-slave-t6c1 krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:50:30.313958 15246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f2bab8b9b2417da3cc4dc8dc562656" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45975 } }
I20260929 02:50:30.314288 15246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:30.314395 15246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f2bab8b9b2417da3cc4dc8dc562656, State: Initialized, Role: FOLLOWER
I20260929 02:50:30.315047 15246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f2bab8b9b2417da3cc4dc8dc562656" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45975 } }
I20260929 02:50:30.315263 15246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:50:30.315395 15246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:50:30.315555 15246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:30.316506 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:30.297598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.4.197.190:45975)
0929 02:50:30.298315 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:30.298327 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:30.298366 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:50:30.300189 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:30.300211 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:30.300596 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:30.300919 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:30.302648 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:30.303990 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:30.305184 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:30.305820 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:30.305975 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:30.309750 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:30.309757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:30.309770 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:30.309805 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:30.312662 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:30.313324 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:30.313329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:30.313335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:30.313489 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:30.313823 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:30.313826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:30.313828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:30.314358 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:30.314640 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:30.315064 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 02:50:30.315476 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":332,"threads_started":1}
I20260929 02:50:30.317794 15246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f2bab8b9b2417da3cc4dc8dc562656" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45975 } }
I20260929 02:50:30.318230 15246 leader_election.cc:302] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54f2bab8b9b2417da3cc4dc8dc562656; no voters:
I20260929 02:50:30.318813 15246 leader_election.cc:288] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:50:30.318962 15251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:30.319561 15251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [term 1 LEADER]: Becoming Leader. State: Replica: 54f2bab8b9b2417da3cc4dc8dc562656, State: Running, Role: LEADER
I20260929 02:50:30.320083 15251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f2bab8b9b2417da3cc4dc8dc562656" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45975 } }
I20260929 02:50:30.320847 15246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:50:30.322759 15253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f2bab8b9b2417da3cc4dc8dc562656. Latest consensus state: current_term: 1 leader_uuid: "54f2bab8b9b2417da3cc4dc8dc562656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f2bab8b9b2417da3cc4dc8dc562656" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45975 } } }
I20260929 02:50:30.323163 15253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:30.323063 15252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54f2bab8b9b2417da3cc4dc8dc562656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f2bab8b9b2417da3cc4dc8dc562656" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45975 } } }
I20260929 02:50:30.323395 15252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:30.324472 15259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:50:30.331027 15259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:50:30.339861 15259 catalog_manager.cc:1347] Generated new cluster ID: 577a83da739142c4ae96ae441eecbb72
I20260929 02:50:30.339989 15259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:50:30.360052 15259 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:50:30.361562 15259 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:50:30.375727 15259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656: Generated new TSK 0
I20260929 02:50:30.376569 15259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:50:30.428529 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45975
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:30.746193 15272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:30.746479 15272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:30.746572 15272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:30.754187 15272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:30.754303 15272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:30.754423 15272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:50:30.764443 15272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:30.765746 15272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:30.767052 15272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:30.779805 15277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:30.779807 15278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:30.779805 15280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:30.780961 15272 server_base.cc:1034] running on GCE node
I20260929 02:50:30.781527 15272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:30.782442 15272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:30.783646 15272 hybrid_clock.cc:648] HybridClock initialized: now 1790650230783610 us; error 28 us; skew 500 ppm
I20260929 02:50:30.783936 15272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:30 dist-test-slave-t6c1 krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:30.786851 15272 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:50:30.788300 15272 webserver.cc:460] Webserver started at http://127.4.197.129:40733/ using document root <none> and password file <none>
I20260929 02:50:30.788946 15272 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:30.789049 15272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:30.789332 15272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:30.792167 15272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "dfb9d67127d94d52a4793b97a2d0c28e"
format_stamp: "Formatted at 2026-09-29 02:50:30 on dist-test-slave-t6c1"
server_key: "8c31f01105ab3e899e567eb679b72bfd"
server_key_iv: "28c4b145fcb4de744e1009eb47a1419a"
server_key_version: "encryptionkey@0"
I20260929 02:50:30.792862 15272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "dfb9d67127d94d52a4793b97a2d0c28e"
format_stamp: "Formatted at 2026-09-29 02:50:30 on dist-test-slave-t6c1"
server_key: "8c31f01105ab3e899e567eb679b72bfd"
server_key_iv: "28c4b145fcb4de744e1009eb47a1419a"
server_key_version: "encryptionkey@0"
I20260929 02:50:30.797547 15272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 02:50:30.800720 15287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:30.801892 15272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:50:30.802023 15272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "dfb9d67127d94d52a4793b97a2d0c28e"
format_stamp: "Formatted at 2026-09-29 02:50:30 on dist-test-slave-t6c1"
server_key: "8c31f01105ab3e899e567eb679b72bfd"
server_key_iv: "28c4b145fcb4de744e1009eb47a1419a"
server_key_version: "encryptionkey@0"
I20260929 02:50:30.802189 15272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:30.832957 15272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:30.837355 15272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:30.837633 15272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:30.838640 15272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:30.840025 15272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:30.840097 15272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:30.840212 15272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:30.840269 15272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:30.869705 15272 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:41213
I20260929 02:50:30.869740 15399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:41213 every 8 connection(s)
I20260929 02:50:30.871001 15272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:50:30.879781 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 15272
I20260929 02:50:30.880012 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:50:30.880623 4886 external_mini_cluster.cc:1428] Setting key a61bda3b2f8114a3b47c549c539d01d7
Sep 29 02:50:30 dist-test-slave-t6c1 krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:50:30.890726 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:30.873583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:58027 (local address 127.4.197.190:45975)
0929 02:50:30.874009 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:30.874014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:30.875118 (+ 1104us) server_negotiation.cc:407] Connection header received
0929 02:50:30.877068 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:30.877072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:30.877153 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:30.877252 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:30.878853 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:30.879844 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:30.881575 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:30.882017 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:30.882135 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:30.885444 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:30.885450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:30.885453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:30.885489 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:30.887807 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:30.888494 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:30.888498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:30.888500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:30.888619 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:30.889024 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:30.889029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:30.889032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:30.889309 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:30.889437 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:30.890151 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 02:50:30.890371 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260929 02:50:30.891732 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:30.874198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45975 (local address 127.4.197.129:58027)
0929 02:50:30.874953 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:30.875002 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:30.876785 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:30.877469 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:30.877478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:30.878038 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:30.878676 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:30.878695 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:30.880013 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:30.880017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:30.881435 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:30.881447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:30.882249 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:30.882256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:30.882491 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:30.883127 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:30.883150 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:30.885213 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:30.887939 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:30.887948 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:30.887966 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:30.888381 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:30.888721 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:30.888726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:30.888730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:30.888921 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:30.889426 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:30.889434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:30.889785 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 02:50:30.890028 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 02:50:30.890569 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":257,"threads_started":1}
I20260929 02:50:30.893226 15400 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45975
I20260929 02:50:30.893546 15400 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:30.894359 15400 heartbeater.cc:511] Master 127.4.197.190:45975 requested a full tablet report, sending...
I20260929 02:50:30.896744 15211 ts_manager.cc:194] Registered new tserver with Master: dfb9d67127d94d52a4793b97a2d0c28e (127.4.197.129:41213)
I20260929 02:50:30.898715 15211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:58027
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:50:30.928241 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45975
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:31.248325 15406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:31.248612 15406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:31.248713 15406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:31.255913 15406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:31.256026 15406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:31.256148 15406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:50:31.266220 15406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:31.267527 15406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:31.268740 15406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:31.281731 15411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:31.281705 15412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:31.281651 15414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:31.282526 15406 server_base.cc:1034] running on GCE node
I20260929 02:50:31.283222 15406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:31.284324 15406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:31.285552 15406 hybrid_clock.cc:648] HybridClock initialized: now 1790650231285494 us; error 61 us; skew 500 ppm
I20260929 02:50:31.285897 15406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:31 dist-test-slave-t6c1 krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650231, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:31.289220 15406 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:50:31.290805 15406 webserver.cc:460] Webserver started at http://127.4.197.130:35109/ using document root <none> and password file <none>
I20260929 02:50:31.291514 15406 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:31.291640 15406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:31.291916 15406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:31.294835 15406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "3b2d6890fcb44a39a8156c240edde492"
format_stamp: "Formatted at 2026-09-29 02:50:31 on dist-test-slave-t6c1"
server_key: "68460af3a53966a00952aa1ad8245a8e"
server_key_iv: "801b97c8070803c3832504d97a742973"
server_key_version: "encryptionkey@0"
I20260929 02:50:31.295642 15406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "3b2d6890fcb44a39a8156c240edde492"
format_stamp: "Formatted at 2026-09-29 02:50:31 on dist-test-slave-t6c1"
server_key: "68460af3a53966a00952aa1ad8245a8e"
server_key_iv: "801b97c8070803c3832504d97a742973"
server_key_version: "encryptionkey@0"
I20260929 02:50:31.300734 15406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:50:31.304247 15421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:31.305781 15406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:50:31.305936 15406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "3b2d6890fcb44a39a8156c240edde492"
format_stamp: "Formatted at 2026-09-29 02:50:31 on dist-test-slave-t6c1"
server_key: "68460af3a53966a00952aa1ad8245a8e"
server_key_iv: "801b97c8070803c3832504d97a742973"
server_key_version: "encryptionkey@0"
I20260929 02:50:31.306126 15406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:31.329268 15406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:31.333699 15406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:31.333973 15406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:31.334964 15406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:31.336352 15406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:31.336432 15406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:31.336558 15406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:31.336617 15406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:31.366358 15406 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:40209
I20260929 02:50:31.366407 15533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:40209 every 8 connection(s)
I20260929 02:50:31.367703 15406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:50:31.369136 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 15406
I20260929 02:50:31.369314 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:50:31.369815 4886 external_mini_cluster.cc:1428] Setting key 426c20d98f134c8a23788030f20e70a4
Sep 29 02:50:31 dist-test-slave-t6c1 krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650231, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:50:31.389518 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.370246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:42397 (local address 127.4.197.190:45975)
0929 02:50:31.370430 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:31.370436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:31.371776 (+ 1340us) server_negotiation.cc:407] Connection header received
0929 02:50:31.374277 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:31.374281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:31.374362 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:31.374448 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:31.376456 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.377145 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.379261 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.379764 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.379875 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.383464 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:31.383470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:31.383475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:31.383516 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:31.386454 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:31.387249 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:31.387254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:31.387257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:31.387423 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:31.387823 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:31.387828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:31.387831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:31.388144 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:31.388270 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:31.388935 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 02:50:31.389177 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:50:31.390460 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.370806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45975 (local address 127.4.197.130:42397)
0929 02:50:31.371623 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:31.371676 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:31.373943 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:31.374727 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:31.374740 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:31.375494 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:31.376267 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:31.376290 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.377284 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.377289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:31.379127 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:31.379141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.379988 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.379994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:31.380279 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.381012 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:31.381037 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:31.383208 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:31.386610 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:31.386617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:31.386631 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:31.387130 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:31.387536 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:31.387539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:31.387541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:31.387722 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:31.388241 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:31.388247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:31.388593 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 02:50:31.388820 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 02:50:31.389266 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":299,"threads_started":1}
I20260929 02:50:31.391834 15534 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45975
I20260929 02:50:31.392117 15534 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:31.392866 15534 heartbeater.cc:511] Master 127.4.197.190:45975 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
I20260929 02:50:31.394218 15211 ts_manager.cc:194] Registered new tserver with Master: 3b2d6890fcb44a39a8156c240edde492 (127.4.197.130:40209)
I20260929 02:50:31.395083 15211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:42397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:50:31.419836 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45975
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:31.737946 15539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:31.738226 15539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:31.738327 15539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:31.745636 15539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:31.745743 15539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:31.745859 15539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:50:31.755621 15539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:50:31.757169 15539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:31.758711 15539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:31.772506 15545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:31.772420 15547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:31.772550 15544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:31.773227 15539 server_base.cc:1034] running on GCE node
I20260929 02:50:31.773904 15539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:31.775034 15539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:31.776279 15539 hybrid_clock.cc:648] HybridClock initialized: now 1790650231776221 us; error 45 us; skew 500 ppm
I20260929 02:50:31.776600 15539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:31 dist-test-slave-t6c1 krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650231, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:31.779764 15539 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:50:31.781137 15539 webserver.cc:460] Webserver started at http://127.4.197.131:46435/ using document root <none> and password file <none>
I20260929 02:50:31.781792 15539 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:31.781903 15539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:31.782202 15539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:31.785044 15539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7"
format_stamp: "Formatted at 2026-09-29 02:50:31 on dist-test-slave-t6c1"
server_key: "3b4eb849165ccdb37e15b5eb31cbf873"
server_key_iv: "045edc5ccf4fb1adb1ab212eeeb6c0e6"
server_key_version: "encryptionkey@0"
I20260929 02:50:31.785787 15539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7"
format_stamp: "Formatted at 2026-09-29 02:50:31 on dist-test-slave-t6c1"
server_key: "3b4eb849165ccdb37e15b5eb31cbf873"
server_key_iv: "045edc5ccf4fb1adb1ab212eeeb6c0e6"
server_key_version: "encryptionkey@0"
I20260929 02:50:31.790723 15539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:50:31.793947 15554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:31.795291 15539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:50:31.795431 15539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7"
format_stamp: "Formatted at 2026-09-29 02:50:31 on dist-test-slave-t6c1"
server_key: "3b4eb849165ccdb37e15b5eb31cbf873"
server_key_iv: "045edc5ccf4fb1adb1ab212eeeb6c0e6"
server_key_version: "encryptionkey@0"
I20260929 02:50:31.795606 15539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:31.810015 15539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:31.813571 15539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:31.813807 15539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:31.814601 15539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:31.815807 15539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:31.815878 15539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:31.815989 15539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:31.816049 15539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:31.844937 15539 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:38005
I20260929 02:50:31.844980 15666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:38005 every 8 connection(s)
I20260929 02:50:31.846164 15539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:50:31.849843 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 15539
I20260929 02:50:31.850052 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:50:31.850596 4886 external_mini_cluster.cc:1428] Setting key 116492633c76e799543f9fc11be1d259
Sep 29 02:50:31 dist-test-slave-t6c1 krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650231, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:50:31.864884 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.848575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:46681 (local address 127.4.197.190:45975)
0929 02:50:31.848771 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:31.848777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:31.850077 (+ 1300us) server_negotiation.cc:407] Connection header received
0929 02:50:31.852335 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:31.852340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:31.852442 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:31.852522 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:31.854112 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.854885 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.856441 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.856851 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.856940 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.860025 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:31.860029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:31.860033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:31.860062 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:31.862254 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:31.862836 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:31.862839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:31.862841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:31.862980 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:31.863307 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:31.863310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:31.863312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:31.863544 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:31.863654 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:31.864387 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 02:50:31.864567 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:50:31.865900 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.849095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45975 (local address 127.4.197.131:46681)
0929 02:50:31.849894 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:31.849944 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:31.852042 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:31.852765 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:31.852774 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:31.853341 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:31.853958 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:31.853975 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.855040 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.855044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:31.856321 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:31.856332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.857022 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.857027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:31.857228 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.857819 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:31.857839 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:31.859858 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:31.862373 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:31.862379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:31.862390 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:31.862753 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:31.863058 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:31.863062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:31.863064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:31.863232 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:31.863650 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:31.863657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:31.864010 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 02:50:31.864286 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 02:50:31.864747 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":275,"threads_started":1}
I20260929 02:50:31.867100 15667 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45975
I20260929 02:50:31.867337 15667 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:31.867971 15667 heartbeater.cc:511] Master 127.4.197.190:45975 requested a full tablet report, sending...
I20260929 02:50:31.869342 15211 ts_manager.cc:194] Registered new tserver with Master: 22f9ae1ebec447c3bdd3c79a53f6b6d7 (127.4.197.131:38005)
I20260929 02:50:31.870141 15211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:46681
I20260929 02:50:31.877615 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:50:31.893494 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.881956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.4.197.190:45975)
0929 02:50:31.882112 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:31.882117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:31.882561 (+ 444us) server_negotiation.cc:407] Connection header received
0929 02:50:31.882706 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:31.882711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:31.882772 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:31.882850 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:31.884226 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.884975 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.886192 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.886668 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.886790 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.889022 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:31.889026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:31.889030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:31.889061 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:31.891495 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:31.892006 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:31.892009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:31.892012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:31.892135 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:31.892408 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:31.892411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:31.892412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:31.892712 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:31.892884 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:31.893104 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 02:50:31.893252 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:50:31.902247 15400 heartbeater.cc:503] Master 127.4.197.190:45975 was elected leader, sending a full tablet report...
I20260929 02:50:31.897284 15211 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:50:31.904292 15211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:50:31.936870 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.915719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.4.197.130:40209)
0929 02:50:31.928785 (+ 13066us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:31.928795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:31.928828 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:50:31.928935 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:31.928944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:31.929233 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:31.929433 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:31.930195 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.931745 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.934341 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.935808 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.935948 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.936153 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:31.936438 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 02:50:31.936578 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12863,"thread_start_us":12731,"threads_started":1}
I20260929 02:50:31.937057 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.915548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:40209 (local address 127.0.0.1:55568)
0929 02:50:31.927815 (+ 12267us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:31.927873 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:31.928103 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:31.929581 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:31.929589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:31.929618 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:31.929993 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:31.930019 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.931857 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.931863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:31.934187 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:31.934198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.936004 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.936010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:31.936214 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.936265 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 02:50:31.936399 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:50:31.936582 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11947,"mutex_wait_us":6,"thread_start_us":13499,"threads_started":1}
I20260929 02:50:31.937706 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.929719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.4.197.131:38005)
0929 02:50:31.930283 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:31.930290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:31.930991 (+ 701us) server_negotiation.cc:407] Connection header received
0929 02:50:31.931165 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:31.931173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:31.931381 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:31.931561 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:31.932170 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.933718 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.935536 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.936717 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.936837 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.937043 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:31.937309 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 02:50:31.937470 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":278,"threads_started":1}
I20260929 02:50:31.938221 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.930303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:38005 (local address 127.0.0.1:35550)
0929 02:50:31.930869 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:31.930885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:31.931030 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:31.931558 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:31.931563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:31.931586 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:31.931965 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:31.931976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.933860 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.933865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:31.935394 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:31.935405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.936923 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.936929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:31.937105 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.937136 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:50:31.937889 (+ 753us) client_negotiation.cc:240] Negotiation successful
0929 02:50:31.938006 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":307,"threads_started":1}
I20260929 02:50:31.938419 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.916029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.4.197.129:41213)
0929 02:50:31.926906 (+ 10877us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:31.926936 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:31.929410 (+ 2474us) server_negotiation.cc:407] Connection header received
0929 02:50:31.929550 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:31.929558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:31.929809 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:31.930006 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:31.930943 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.932227 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.934458 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.935808 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.935948 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.936176 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:31.937945 (+ 1769us) server_negotiation.cc:299] Negotiation successful
0929 02:50:31.938116 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10640,"thread_start_us":10923,"threads_started":1}
I20260929 02:50:31.938678 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:31.915695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41213 (local address 127.0.0.1:59054)
0929 02:50:31.929272 (+ 13577us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:31.929287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:31.929408 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:31.930233 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:31.930237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:31.930262 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:31.930710 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:31.930722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.932367 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.932372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:31.934303 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:31.934315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:31.937208 (+ 2893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:31.937213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:31.937360 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:31.937389 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:50:31.937464 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 02:50:31.937582 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13465,"thread_start_us":12667,"threads_started":1}
I20260929 02:50:31.941382 15469 tablet_service.cc:1467] Processing CreateTablet for tablet ae2f0a5879bd4ce5b7992709f9498eb3 (DEFAULT_TABLE table=test-table [id=2872be799b2e4aa684ebf488631a360a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:31.941779 15602 tablet_service.cc:1467] Processing CreateTablet for tablet ae2f0a5879bd4ce5b7992709f9498eb3 (DEFAULT_TABLE table=test-table [id=2872be799b2e4aa684ebf488631a360a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:31.942603 15469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2f0a5879bd4ce5b7992709f9498eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:31.942147 15335 tablet_service.cc:1467] Processing CreateTablet for tablet ae2f0a5879bd4ce5b7992709f9498eb3 (DEFAULT_TABLE table=test-table [id=2872be799b2e4aa684ebf488631a360a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:31.943264 15602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2f0a5879bd4ce5b7992709f9498eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:31.943420 15335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2f0a5879bd4ce5b7992709f9498eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:31.953434 15685 tablet_bootstrap.cc:492] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: Bootstrap starting.
I20260929 02:50:31.957167 15686 tablet_bootstrap.cc:492] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: Bootstrap starting.
I20260929 02:50:31.957588 15685 tablet_bootstrap.cc:654] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:31.958899 15685 log.cc:824] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:31.960381 15687 tablet_bootstrap.cc:492] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: Bootstrap starting.
I20260929 02:50:31.961047 15686 tablet_bootstrap.cc:654] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:31.961993 15685 tablet_bootstrap.cc:492] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: No bootstrap required, opened a new log
I20260929 02:50:31.962263 15685 ts_tablet_manager.cc:1397] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 02:50:31.962373 15686 log.cc:824] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:31.963692 15687 tablet_bootstrap.cc:654] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:31.964847 15687 log.cc:824] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:31.965505 15686 tablet_bootstrap.cc:492] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: No bootstrap required, opened a new log
I20260929 02:50:31.965839 15686 ts_tablet_manager.cc:1397] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260929 02:50:31.967789 15687 tablet_bootstrap.cc:492] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: No bootstrap required, opened a new log
I20260929 02:50:31.968077 15687 ts_tablet_manager.cc:1397] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:50:31.971705 15685 raft_consensus.cc:348] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } }
I20260929 02:50:31.972216 15685 raft_consensus.cc:374] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:31.972362 15685 raft_consensus.cc:729] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2d6890fcb44a39a8156c240edde492, State: Initialized, Role: FOLLOWER
I20260929 02:50:31.973042 15685 consensus_queue.cc:260] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } }
I20260929 02:50:31.973296 15686 raft_consensus.cc:348] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } }
I20260929 02:50:31.973801 15686 raft_consensus.cc:374] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:31.973933 15686 raft_consensus.cc:729] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb9d67127d94d52a4793b97a2d0c28e, State: Initialized, Role: FOLLOWER
I20260929 02:50:31.974190 15534 heartbeater.cc:503] Master 127.4.197.190:45975 was elected leader, sending a full tablet report...
I20260929 02:50:31.974877 15686 consensus_queue.cc:260] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } }
I20260929 02:50:31.975078 15685 ts_tablet_manager.cc:1428] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260929 02:50:31.976894 15686 ts_tablet_manager.cc:1428] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260929 02:50:31.977576 15687 raft_consensus.cc:348] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } }
I20260929 02:50:31.977943 15687 raft_consensus.cc:374] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:31.978048 15687 raft_consensus.cc:729] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f9ae1ebec447c3bdd3c79a53f6b6d7, State: Initialized, Role: FOLLOWER
I20260929 02:50:31.978622 15687 consensus_queue.cc:260] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } }
I20260929 02:50:31.981086 15667 heartbeater.cc:503] Master 127.4.197.190:45975 was elected leader, sending a full tablet report...
I20260929 02:50:31.981503 15687 ts_tablet_manager.cc:1428] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 02:50:32.093202 15692 raft_consensus.cc:482] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:50:32.093492 15692 raft_consensus.cc:504] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } }
I20260929 02:50:32.095171 15692 leader_election.cc:288] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b2d6890fcb44a39a8156c240edde492 (127.4.197.130:40209), 22f9ae1ebec447c3bdd3c79a53f6b6d7 (127.4.197.131:38005)
I20260929 02:50:32.097604 15691 raft_consensus.cc:482] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 02:50:32.097651 15668 tablet.cc:2369] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:50:32.097903 15691 raft_consensus.cc:504] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } }
I20260929 02:50:32.099699 15691 leader_election.cc:288] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfb9d67127d94d52a4793b97a2d0c28e (127.4.197.129:41213), 22f9ae1ebec447c3bdd3c79a53f6b6d7 (127.4.197.131:38005)
I20260929 02:50:32.103902 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.096305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:38005 (local address 127.4.197.129:51853)
0929 02:50:32.096919 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:32.096935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:32.097053 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:32.097375 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:32.097379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:32.097400 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:32.097726 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:32.097736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.099111 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.099116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:32.101410 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:32.101426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.103073 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.103079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:32.103229 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.103260 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:50:32.103338 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:50:32.103457 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":1,"thread_start_us":313,"threads_started":1}
I20260929 02:50:32.104487 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.096412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:51853 (local address 127.4.197.131:38005)
0929 02:50:32.096673 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:32.096679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:32.097012 (+ 333us) server_negotiation.cc:407] Connection header received
0929 02:50:32.097197 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:32.097202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:32.097269 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:32.097350 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:32.097889 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.098984 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.101556 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.102703 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.102851 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.102943 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:32.104163 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0929 02:50:32.104284 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:50:32.104681 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.095587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:34677 (local address 127.4.197.130:40209)
0929 02:50:32.095757 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:32.095762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:32.096249 (+ 487us) server_negotiation.cc:407] Connection header received
0929 02:50:32.096480 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:32.096486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:32.096565 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:32.096656 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:32.097234 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.098312 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.101797 (+ 3485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.103132 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.103277 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.103347 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:32.104347 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0929 02:50:32.104462 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:50:32.104259 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.095522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:40209 (local address 127.4.197.129:34677)
0929 02:50:32.096157 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:32.096175 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:32.096311 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:32.096761 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:32.096766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:32.096795 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:32.097094 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:32.097104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.098459 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.098464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:32.101679 (+ 3215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:32.101689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.103485 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.103490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:32.103661 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.103693 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:50:32.103781 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:50:32.103909 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":1197,"thread_start_us":331,"threads_started":1}
I20260929 02:50:32.105847 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2f0a5879bd4ce5b7992709f9498eb3" candidate_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" is_pre_election: true
I20260929 02:50:32.105839 15489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2f0a5879bd4ce5b7992709f9498eb3" candidate_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b2d6890fcb44a39a8156c240edde492" is_pre_election: true
I20260929 02:50:32.106302 15489 raft_consensus.cc:2454] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb9d67127d94d52a4793b97a2d0c28e in term 0.
I20260929 02:50:32.106341 15622 raft_consensus.cc:2454] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb9d67127d94d52a4793b97a2d0c28e in term 0.
I20260929 02:50:32.107013 15288 leader_election.cc:302] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22f9ae1ebec447c3bdd3c79a53f6b6d7, dfb9d67127d94d52a4793b97a2d0c28e; no voters:
I20260929 02:50:32.107438 15692 raft_consensus.cc:2781] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:50:32.107586 15692 raft_consensus.cc:482] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:50:32.107671 15692 raft_consensus.cc:3037] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:32.110684 15692 raft_consensus.cc:504] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } }
I20260929 02:50:32.110841 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.100147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:33251 (local address 127.4.197.129:41213)
0929 02:50:32.100297 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:32.100302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:32.102380 (+ 2078us) server_negotiation.cc:407] Connection header received
0929 02:50:32.102610 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:32.102615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:32.102678 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:32.102732 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:32.103569 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.104665 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.108983 (+ 4318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.109956 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.110106 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.110174 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:32.110480 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 02:50:32.110600 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:50:32.111074 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.099824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41213 (local address 127.4.197.130:33251)
0929 02:50:32.102281 (+ 2457us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:32.102302 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:32.102473 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:32.102907 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:32.102935 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:32.102961 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:32.103408 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:32.103419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.104894 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.104899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:32.108802 (+ 3903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:32.108813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.110180 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.110187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:32.110346 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.110370 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:50:32.110457 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 02:50:32.110634 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2290,"mutex_wait_us":19,"thread_start_us":5619,"threads_started":1}
I20260929 02:50:32.111357 15692 leader_election.cc:288] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [CANDIDATE]: Term 1 election: Requested vote from peers 3b2d6890fcb44a39a8156c240edde492 (127.4.197.130:40209), 22f9ae1ebec447c3bdd3c79a53f6b6d7 (127.4.197.131:38005)
I20260929 02:50:32.111806 15489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2f0a5879bd4ce5b7992709f9498eb3" candidate_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b2d6890fcb44a39a8156c240edde492"
I20260929 02:50:32.111841 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2f0a5879bd4ce5b7992709f9498eb3" candidate_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7"
I20260929 02:50:32.112002 15489 raft_consensus.cc:3037] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:32.112021 15622 raft_consensus.cc:3037] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:32.111899 15355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2f0a5879bd4ce5b7992709f9498eb3" candidate_uuid: "3b2d6890fcb44a39a8156c240edde492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" is_pre_election: true
I20260929 02:50:32.112308 15355 raft_consensus.cc:2379] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b2d6890fcb44a39a8156c240edde492 in current term 1: Already voted for candidate dfb9d67127d94d52a4793b97a2d0c28e in this term.
I20260929 02:50:32.112658 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.100652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:54613 (local address 127.4.197.131:38005)
0929 02:50:32.104909 (+ 4257us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:32.104915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:32.105865 (+ 950us) server_negotiation.cc:407] Connection header received
0929 02:50:32.106106 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:32.106110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:32.106172 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:32.106229 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:32.107183 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.108180 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.110441 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.111590 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.111728 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.111788 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:32.112292 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 02:50:32.112405 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4092,"thread_start_us":6509,"threads_started":1}
I20260929 02:50:32.113389 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.100176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:38005 (local address 127.4.197.130:54613)
0929 02:50:32.105777 (+ 5601us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:32.105792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:32.105956 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:32.106472 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:32.106476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:32.106499 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:32.107002 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:32.107015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.108312 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.108317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:32.110298 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:32.110311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.111929 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.111935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:32.112091 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.112137 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 02:50:32.112850 (+ 713us) client_negotiation.cc:240] Negotiation successful
0929 02:50:32.112978 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5449,"mutex_wait_us":1,"thread_start_us":1744,"threads_started":1}
I20260929 02:50:32.114063 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2f0a5879bd4ce5b7992709f9498eb3" candidate_uuid: "3b2d6890fcb44a39a8156c240edde492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" is_pre_election: true
I20260929 02:50:32.114831 15622 raft_consensus.cc:2454] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb9d67127d94d52a4793b97a2d0c28e in term 1.
I20260929 02:50:32.114889 15422 leader_election.cc:302] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b2d6890fcb44a39a8156c240edde492; no voters: 22f9ae1ebec447c3bdd3c79a53f6b6d7, dfb9d67127d94d52a4793b97a2d0c28e
I20260929 02:50:32.115006 15489 raft_consensus.cc:2454] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb9d67127d94d52a4793b97a2d0c28e in term 1.
I20260929 02:50:32.115149 15288 leader_election.cc:302] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22f9ae1ebec447c3bdd3c79a53f6b6d7, dfb9d67127d94d52a4793b97a2d0c28e; no voters:
I20260929 02:50:32.115309 15691 raft_consensus.cc:2726] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:50:32.115427 15692 raft_consensus.cc:2781] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:32.115842 15692 raft_consensus.cc:686] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 1 LEADER]: Becoming Leader. State: Replica: dfb9d67127d94d52a4793b97a2d0c28e, State: Running, Role: LEADER
I20260929 02:50:32.116237 15692 consensus_queue.cc:237] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } }
W20260929 02:50:32.119505 15535 tablet.cc:2369] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:50:32.120381 15211 catalog_manager.cc:5495] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e reported cstate change: term changed from 0 to 1, leader changed from <none> to dfb9d67127d94d52a4793b97a2d0c28e (127.4.197.129). New cstate: current_term: 1 leader_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb9d67127d94d52a4793b97a2d0c28e" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 } health_report { overall_health: UNKNOWN } } }
W20260929 02:50:32.123123 15401 tablet.cc:2369] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:50:32.165483 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.160157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.4.197.129:41213)
0929 02:50:32.160334 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:32.160340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:32.160360 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:50:32.160536 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:32.160541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:32.160627 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:32.160753 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:32.161125 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.161871 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.163334 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.163759 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.163868 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.164365 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:32.164891 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:32.165024 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:32.165117 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 02:50:32.165204 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:50:32.173575 15622 raft_consensus.cc:1261] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7 [term 1 FOLLOWER]: Refusing update from remote peer dfb9d67127d94d52a4793b97a2d0c28e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:32.173648 15489 raft_consensus.cc:1261] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 1 FOLLOWER]: Refusing update from remote peer dfb9d67127d94d52a4793b97a2d0c28e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:32.174396 15699 consensus_queue.cc:1035] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "22f9ae1ebec447c3bdd3c79a53f6b6d7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:32.174829 15692 consensus_queue.cc:1035] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b2d6890fcb44a39a8156c240edde492" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:32.187810 15704 mvcc.cc:205] Tried to move back new op lower bound from 7334503350973509632 to 7334503350751145984. Current Snapshot: MvccSnapshot[applied={T|T < 7334503350973509632}]
I20260929 02:50:32.198907 15210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:32.199252 15210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:32.202708 15210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54f2bab8b9b2417da3cc4dc8dc562656: Sending DeleteTablet for 3 replicas of tablet ae2f0a5879bd4ce5b7992709f9498eb3
I20260929 02:50:32.203709 15335 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2f0a5879bd4ce5b7992709f9498eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:32 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:59054
I20260929 02:50:32.203850 15469 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2f0a5879bd4ce5b7992709f9498eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:32 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:55568
I20260929 02:50:32.203984 15602 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2f0a5879bd4ce5b7992709f9498eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:32 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:35550
I20260929 02:50:32.204493 15712 tablet_replica.cc:330] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: stopping tablet replica
I20260929 02:50:32.204852 15712 raft_consensus.cc:2229] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:50:32.205324 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 15272
I20260929 02:50:32.205403 15712 raft_consensus.cc:2258] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:32.214704 15714 tablet_replica.cc:330] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: stopping tablet replica
I20260929 02:50:32.215112 15714 raft_consensus.cc:2229] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:32.215610 15714 raft_consensus.cc:2258] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:32.216427 15712 ts_tablet_manager.cc:1905] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:50:32 dist-test-slave-t6c1 krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:50:32.217589 15714 ts_tablet_manager.cc:1905] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:32.220166 15713 tablet_replica.cc:330] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: stopping tablet replica
I20260929 02:50:32.220604 15713 raft_consensus.cc:2229] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:32.221081 15713 raft_consensus.cc:2258] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:32.223655 15714 ts_tablet_manager.cc:1918] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:32.223802 15714 log.cc:1196] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/ae2f0a5879bd4ce5b7992709f9498eb3
I20260929 02:50:32.223801 15712 ts_tablet_manager.cc:1918] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:32.223942 15712 log.cc:1196] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/ae2f0a5879bd4ce5b7992709f9498eb3
I20260929 02:50:32.224336 15714 ts_tablet_manager.cc:1939] T ae2f0a5879bd4ce5b7992709f9498eb3 P 22f9ae1ebec447c3bdd3c79a53f6b6d7: Deleting consensus metadata
I20260929 02:50:32.224375 15713 ts_tablet_manager.cc:1905] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:32.224465 15712 ts_tablet_manager.cc:1939] T ae2f0a5879bd4ce5b7992709f9498eb3 P dfb9d67127d94d52a4793b97a2d0c28e: Deleting consensus metadata
I20260929 02:50:32.225891 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.207285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.4.197.129:41213)
0929 02:50:32.207423 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:32.207429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:32.208029 (+ 600us) server_negotiation.cc:407] Connection header received
0929 02:50:32.208186 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:32.208190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:32.208269 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:32.208361 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:32.210060 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.211218 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.213059 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.213692 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.213797 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.218524 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:32.218536 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:32.218550 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:32.218595 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:32.222288 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:32.223312 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:32.223320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:32.223324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:32.223460 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:32.223920 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:32.223925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:32.223932 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:32.224458 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:32.224761 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:32.225345 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 02:50:32.225562 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 02:50:32.226078 15195 catalog_manager.cc:4841] TS 22f9ae1ebec447c3bdd3c79a53f6b6d7 (127.4.197.131:38005): tablet ae2f0a5879bd4ce5b7992709f9498eb3 (table test-table [id=2872be799b2e4aa684ebf488631a360a]) successfully deleted
I20260929 02:50:32.227284 15395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:32.227257 15197 catalog_manager.cc:4841] TS dfb9d67127d94d52a4793b97a2d0c28e (127.4.197.129:41213): tablet ae2f0a5879bd4ce5b7992709f9498eb3 (table test-table [id=2872be799b2e4aa684ebf488631a360a]) successfully deleted
I20260929 02:50:32.231899 15713 ts_tablet_manager.cc:1918] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:32.232036 15713 log.cc:1196] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/ae2f0a5879bd4ce5b7992709f9498eb3
I20260929 02:50:32.232425 15713 ts_tablet_manager.cc:1939] T ae2f0a5879bd4ce5b7992709f9498eb3 P 3b2d6890fcb44a39a8156c240edde492: Deleting consensus metadata
I20260929 02:50:32.233819 15195 catalog_manager.cc:4841] TS 3b2d6890fcb44a39a8156c240edde492 (127.4.197.130:40209): tablet ae2f0a5879bd4ce5b7992709f9498eb3 (table test-table [id=2872be799b2e4aa684ebf488631a360a]) successfully deleted
I20260929 02:50:32.329603 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 15272
I20260929 02:50:32.343621 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 15406
Sep 29 02:50:32 dist-test-slave-t6c1 krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:50:32.357115 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.344295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.4.197.130:40209)
0929 02:50:32.344466 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:32.344470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:32.344488 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:50:32.344623 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:32.344627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:32.344698 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:32.344776 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:32.345863 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.346543 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.347713 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.348090 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.348177 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.351880 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:32.351893 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:32.351908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:32.351946 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:32.354483 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:32.355245 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:32.355252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:32.355255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:32.355422 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:32.355737 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:32.355740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:32.355742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:32.356148 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:32.356361 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:32.356593 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 02:50:32.356785 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:50:32.357679 15529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:32.449499 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 15406
I20260929 02:50:32.462455 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 15539
Sep 29 02:50:32 dist-test-slave-t6c1 krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:50:32.476853 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.463169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.4.197.131:38005)
0929 02:50:32.463355 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:32.463359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:32.463377 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:50:32.463477 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:32.463482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:32.463545 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:32.463648 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:32.464825 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.465483 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.466662 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:32.467190 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:32.467284 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:32.471627 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:32.471645 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:32.471662 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:32.471705 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:32.474441 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:32.475167 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:32.475174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:32.475177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:32.475318 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:32.475649 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:32.475652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:32.475654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:32.475985 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:32.476199 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:32.476401 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 02:50:32.476556 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:50:32.477424 15662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:32.563421 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 15539
I20260929 02:50:32.576139 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 15179
I20260929 02:50:32.577229 15241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:32.683454 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 15179
2026-09-29T02:50:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4040 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:50:32 dist-test-slave-t6c1 krb5kdc[15722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:50:32 dist-test-slave-t6c1 krb5kdc[15722](info): set up 2 sockets
Sep 29 02:50:32 dist-test-slave-t6c1 krb5kdc[15722](info): commencing operation
krb5kdc: starting...
W20260929 02:50:33.687642 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:50:33 dist-test-slave-t6c1 krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:50:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.197.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.197.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/rangeradmin_127.4.197.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/rangeradmin_127.4.197.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.197.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.197.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/rangerlookup_127.4.197.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/rangerlookup_127.4.197.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.197.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.197.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/HTTP_127.4.197.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/HTTP_127.4.197.148@KRBTEST.COM.keytab.
I20260929 02:50:33.991101 4886 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgLoZGx/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/postgres -l logfile start
2026-09-29 02:50:36.300 UTC [15752] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 02:50:36.300 UTC [15752] LOG: listening on IPv4 address "127.4.197.148", port 59303
2026-09-29 02:50:36.303 UTC [15752] LOG: listening on Unix socket "/tmp/.s.PGSQL.59303"
2026-09-29 02:50:36.322 UTC [15755] LOG: database system was shut down at 2026-09-29 02:50:35 UTC
2026-09-29 02:50:36.325 UTC [15752] LOG: database system is ready to accept connections
I20260929 02:50:37.207288 4886 mini_postgres.cc:96] Postgres bound to 59303
2026-09-29 02:50:37.222 UTC [15763] FATAL: database "slave" does not exist
127.4.197.148:59303 - accepting connections
I20260929 02:50:37.223407 4886 mini_ranger.cc:161] Starting Ranger...
I20260929 02:50:37.247721 4886 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 02:50:37.687413 4886 mini_ranger.cc:87] Created ranger Postgres database
I20260929 02:50:37.687533 4886 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-admin
2026-09-29 02:50:37,988 [I] DB FLAVOR :POSTGRES
2026-09-29 02:50:37,989 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:50:37,989 [I] Checking connection..
2026-09-29 02:50:37,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 02:50:38,217 [I] Checking connection passed.
2026-09-29 02:50:38,218 [I] --------- Verifying version history table ---------
2026-09-29 02:50:38,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:50:38,491 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:50:38,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:50:38,736 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:50:38,736 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 02:50:38,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 02:50:38.979 UTC [15827] WARNING: there is no transaction in progress
2026-09-29 02:50:38,984 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 02:50:38,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:50:39,241 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 02:50:39,241 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:50:39,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 02:50:39,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 02:50:39,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t6c1.c.gcp-upstream.internal','N') ;"
2026-09-29 02:50:39,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:50:40,220 [I] Table x_portal_user does not exist in database ranger
2026-09-29 02:50:40,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:50:40,459 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 02:50:40,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:50:40,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:50:40,922 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 02:50:40,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 02:50:41.891 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:41.907 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:41.922 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:41.937 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:41.953 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:41.971 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42.027 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42.033 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42.047 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42.057 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42.067 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42.080 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42.090 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42.096 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42.103 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42.109 UTC [15935] WARNING: there is no transaction in progress
2026-09-29 02:50:42,737 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 02:50:42,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 02:50:42,989 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 02:50:42,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:50:43,213 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 02:50:43,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:50:43,465 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 02:50:43,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:50:43,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:50:43,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t6c1.c.gcp-upstream.internal';"
2026-09-29 02:50:44,208 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 02:50:44,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 02:50:44,443 [I] DB_PATCHES have already been applied
I20260929 02:50:44.449122 4886 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgLoZGx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 02:50:44.449218 4886 mini_ranger.cc:193] Using host: 127.4.197.148
I20260929 02:50:44.463691 4886 mini_ranger.cc:227] Ranger admin URL: http://127.4.197.148:55799
Sep 29, 2026 2:50:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:50:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:50:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:50:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:50:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:50:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:50:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 02:50:46.595878 4886 mini_ranger.cc:160] Time spent starting Ranger: real 9.373s user 0.007s sys 0.043s
Sep 29, 2026 2:51:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:51:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 2:51:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:51:06 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:51:07 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 02:51:07.781937 4886 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.197.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.197.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/rangerkms_127.4.197.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/rangerkms_127.4.197.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.197.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.197.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.197.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/HTTP_127.4.197.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.197.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/HTTP_127.4.197.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 02:51:07 dist-test-slave-t6c1 krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650267, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 02:51:07.943951 4886 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 02:51:07.963454 4886 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 02:51:08.790347 4886 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 02:51:08.790475 4886 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-kms
I20260929 02:51:08.790596 4886 mini_ranger_kms.cc:227] Using postgres at 127.4.197.148:59303
2026-09-29 02:51:09,172 [I] DB FLAVOR :POSTGRES
2026-09-29 02:51:09,172 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:51:09,172 [I] Checking connection
2026-09-29 02:51:09,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 02:51:09,395 [I] connection success
2026-09-29 02:51:09,395 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 02:51:09,395 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 02:51:09,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 02:51:09,618 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 02:51:09,618 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:51:09,619 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 02:51:09,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:59303/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 02:51:09,884 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 02:51:10.332914 4886 mini_ranger_kms.cc:310] Created kms service
I20260929 02:51:10.517992 4886 mini_ranger_kms.cc:326] Created kudu user
I20260929 02:51:10.561474 4886 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 02:51:10 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650267, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:51:10.948489 4886 mini_ranger_kms.cc:384] Added ranger policy
I20260929 02:51:10.948853 4886 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 02:51:10.948917 4886 mini_ranger_kms.cc:240] Using host: 127.4.197.148
I20260929 02:51:10.965790 4886 mini_ranger_kms.cc:276] Ranger KMS PID: 16116
I20260929 02:51:10.965996 4886 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.197.148:49761
Sep 29, 2026 2:51:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:51:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:51:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:51:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:51:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:51:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 02:51:13.046502 4886 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.103s user 0.119s sys 0.275s
I20260929 02:51:13.046716 4886 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 02:51:13.046891 4886 mini_ranger_kms.cc:401] 127.4.197.148:49761/kms/v1/keys
Sep 29 02:51:13 dist-test-slave-t6c1 krb5kdc[15722](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650273, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.197.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:51:13 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650273, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.197.148@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
Sep 29 02:51:13 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650273, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.197.148@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
Sep 29 02:51:16 dist-test-slave-t6c1 krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:51:16.307318 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:35221
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:35221
--ranger_config_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:49761/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:16.637089 16167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:16.637454 16167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:16.637576 16167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:16.645148 16167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:16.645267 16167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:16.645354 16167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:16.645427 16167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:16.645491 16167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:16.655236 16167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.197.148:49761/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:35221
--ranger_config_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:35221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:51:16.656469 16167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:16.657701 16167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:16.669382 16172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:16.669589 16173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:16.669538 16175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:16.670584 16167 server_base.cc:1034] running on GCE node
I20260929 02:51:16.671413 16167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:16.672813 16167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:16.674031 16167 hybrid_clock.cc:648] HybridClock initialized: now 1790650276673989 us; error 28 us; skew 500 ppm
I20260929 02:51:16.674410 16167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:16 dist-test-slave-t6c1 krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650276, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:16.682744 16167 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:51:16.684309 16167 webserver.cc:460] Webserver started at http://127.4.197.190:37267/ using document root <none> and password file <none>
I20260929 02:51:16.684949 16167 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:16.685046 16167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:16.685338 16167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:51:16 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650276, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:51:16.746616 16167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "da2188619870417fb1863680eebf2b37"
format_stamp: "Formatted at 2026-09-29 02:51:16 on dist-test-slave-t6c1"
server_key: "cc4e8b21b927ab1bb98abb5db855625f"
server_key_iv: "23b9a5efc0b04b2ac80e086f58eb3ce5"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:16.747610 16167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "da2188619870417fb1863680eebf2b37"
format_stamp: "Formatted at 2026-09-29 02:51:16 on dist-test-slave-t6c1"
server_key: "cc4e8b21b927ab1bb98abb5db855625f"
server_key_iv: "23b9a5efc0b04b2ac80e086f58eb3ce5"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:16.753553 16167 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 02:51:16.775265 16182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:16.777108 16167 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:51:16.777311 16167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "da2188619870417fb1863680eebf2b37"
format_stamp: "Formatted at 2026-09-29 02:51:16 on dist-test-slave-t6c1"
server_key: "cc4e8b21b927ab1bb98abb5db855625f"
server_key_iv: "23b9a5efc0b04b2ac80e086f58eb3ce5"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:16.777582 16167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:16.801081 16167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:16.806152 16167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:16.806474 16167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:16.830736 16167 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:35221
I20260929 02:51:16.830786 16233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:35221 every 8 connection(s)
I20260929 02:51:16.832065 16167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:16.835729 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:16.836477 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 16167
I20260929 02:51:16.836683 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:51:16.837483 4886 external_mini_cluster.cc:1428] Setting key e664a10b930d813193a09177927f4875
I20260929 02:51:16.849390 16234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37: Bootstrap starting.
Sep 29 02:51:16 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:51:16.852145 16234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:16.853323 16234 log.cc:824] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:16.856400 16234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37: No bootstrap required, opened a new log
I20260929 02:51:16.859918 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:16.839448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.4.197.190:35221)
0929 02:51:16.840141 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:16.840155 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:16.840204 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:51:16.842126 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:16.842153 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:16.842659 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:16.843106 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:16.844131 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:16.846088 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:16.847307 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:16.848786 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:16.848959 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:16.852361 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:16.852371 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:16.852388 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:16.852433 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:16.855997 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:16.856603 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:16.856609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:16.856614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:16.856778 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:16.857132 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:16.857135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:16.857138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:16.857573 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:16.857860 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:16.858264 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 02:51:16.858632 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":261,"threads_started":1}
I20260929 02:51:16.865334 16234 raft_consensus.cc:348] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2188619870417fb1863680eebf2b37" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35221 } }
I20260929 02:51:16.865633 16234 raft_consensus.cc:374] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:16.865723 16234 raft_consensus.cc:729] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da2188619870417fb1863680eebf2b37, State: Initialized, Role: FOLLOWER
I20260929 02:51:16.866220 16234 consensus_queue.cc:260] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2188619870417fb1863680eebf2b37" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35221 } }
I20260929 02:51:16.866428 16234 raft_consensus.cc:388] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:16.866549 16234 raft_consensus.cc:482] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:16.866703 16234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:16.868764 16234 raft_consensus.cc:504] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2188619870417fb1863680eebf2b37" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35221 } }
I20260929 02:51:16.869205 16234 leader_election.cc:302] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da2188619870417fb1863680eebf2b37; no voters:
I20260929 02:51:16.869787 16234 leader_election.cc:288] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:16.869911 16239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:16.870774 16239 raft_consensus.cc:686] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [term 1 LEADER]: Becoming Leader. State: Replica: da2188619870417fb1863680eebf2b37, State: Running, Role: LEADER
I20260929 02:51:16.871323 16239 consensus_queue.cc:237] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2188619870417fb1863680eebf2b37" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35221 } }
I20260929 02:51:16.871951 16234 sys_catalog.cc:564] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:16.874639 16240 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da2188619870417fb1863680eebf2b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2188619870417fb1863680eebf2b37" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35221 } } }
I20260929 02:51:16.874673 16241 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da2188619870417fb1863680eebf2b37. Latest consensus state: current_term: 1 leader_uuid: "da2188619870417fb1863680eebf2b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2188619870417fb1863680eebf2b37" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35221 } } }
I20260929 02:51:16.875032 16240 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:16.875053 16241 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:16.880883 16245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:16.886101 16245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:16.886772 16234 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:51:16.902803 16245 catalog_manager.cc:1347] Generated new cluster ID: acac73ecee2e4cb5a98078598c09dcdd
I20260929 02:51:16.902962 16245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:16.934569 16245 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:16.936719 16245 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:16.945508 16245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37: Generated new TSK 0
I20260929 02:51:16.946674 16245 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:51:17.636705 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:49761/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35221
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:51:17 dist-test-slave-t6c1 krb5kdc[15722](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650277, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 02:51:18.047044 16279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:18.047531 16279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:18.047701 16279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:18.059825 16279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:18.060010 16279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:18.060220 16279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:51:18.076720 16279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.197.148:49761/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:51:18.078465 16279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:18.080170 16279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:18.098459 16289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:18.098572 16290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:18.107661 16279 server_base.cc:1034] running on GCE node
W20260929 02:51:18.108350 16292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:18.109140 16279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:18.110318 16279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:18.111552 16279 hybrid_clock.cc:648] HybridClock initialized: now 1790650278111487 us; error 55 us; skew 500 ppm
I20260929 02:51:18.111948 16279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:18 dist-test-slave-t6c1 krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650278, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:18.116699 16279 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:51:18.135349 16279 webserver.cc:460] Webserver started at http://127.4.197.129:36799/ using document root <none> and password file <none>
I20260929 02:51:18.136435 16279 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:18.136584 16279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:18.136936 16279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:51:18 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650278, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:51:18.155716 16279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "368a1795ee5d4258bdd74462416d3934"
format_stamp: "Formatted at 2026-09-29 02:51:18 on dist-test-slave-t6c1"
server_key: "e4619163c783e757694fba40f5ebaa77"
server_key_iv: "d9526947e90d1c5260b457ac0175b198"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:18.158130 16279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "368a1795ee5d4258bdd74462416d3934"
format_stamp: "Formatted at 2026-09-29 02:51:18 on dist-test-slave-t6c1"
server_key: "e4619163c783e757694fba40f5ebaa77"
server_key_iv: "d9526947e90d1c5260b457ac0175b198"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:18.167969 16279 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.003s
I20260929 02:51:18.186573 16299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:18.188548 16279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:51:18.188741 16279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "368a1795ee5d4258bdd74462416d3934"
format_stamp: "Formatted at 2026-09-29 02:51:18 on dist-test-slave-t6c1"
server_key: "e4619163c783e757694fba40f5ebaa77"
server_key_iv: "d9526947e90d1c5260b457ac0175b198"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:18.189006 16279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:18.294327 16279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:18.299551 16279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:18.299880 16279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:18.307160 16279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:18.315518 16279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:18.315630 16279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:18.315755 16279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:18.315817 16279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:51:18 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650277, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:51:18.798290 16279 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:36813
I20260929 02:51:18.798300 16411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:36813 every 8 connection(s)
I20260929 02:51:18.800184 16279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:51:18.804538 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 16279
I20260929 02:51:18.804744 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:51:18.805528 4886 external_mini_cluster.cc:1428] Setting key ce4bbb49eda9cd7d4365906adfc1805d
Sep 29 02:51:18 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650278, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Sep 29 02:51:18 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650277, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:51:18.857211 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:18.807644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:51771 (local address 127.4.197.190:35221)
0929 02:51:18.830664 (+ 23020us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:18.830675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:18.832494 (+ 1819us) server_negotiation.cc:407] Connection header received
0929 02:51:18.834527 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:18.834533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:18.834646 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:18.834749 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:18.837004 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:18.838095 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:18.840561 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:18.841208 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:18.841306 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:18.849802 (+ 8496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:18.849809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:18.849814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:18.849871 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:18.853174 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:18.854099 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:18.854105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:18.854108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:18.854264 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:18.854818 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:18.854823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:18.854826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:18.855182 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:18.855308 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:18.856616 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0929 02:51:18.856833 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22873,"thread_start_us":23039,"threads_started":1}
I20260929 02:51:18.860952 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:18.806989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:35221 (local address 127.4.197.129:51771)
0929 02:51:18.831385 (+ 24396us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:18.831449 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:18.834040 (+ 2591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:18.835261 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:18.835275 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:18.835949 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:18.836796 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:18.836821 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:18.838277 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:18.838282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:18.840383 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:18.840398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:18.841677 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:18.841684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:18.842004 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:18.843620 (+ 1616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:18.843657 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:18.849543 (+ 5886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:18.853343 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:18.853352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:18.853371 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:18.853941 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:18.854406 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:18.854411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:18.854415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:18.854690 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:18.855398 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:18.855406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:18.856169 (+ 763us) client_negotiation.cc:769] Sending connection context
0929 02:51:18.857733 (+ 1564us) client_negotiation.cc:240] Negotiation successful
0929 02:51:18.858262 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24038,"thread_start_us":25416,"threads_started":1}
I20260929 02:51:18.863751 16413 heartbeater.cc:347] Connected to a master server at 127.4.197.190:35221
I20260929 02:51:18.864089 16413 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:18.864912 16413 heartbeater.cc:511] Master 127.4.197.190:35221 requested a full tablet report, sending...
I20260929 02:51:18.870541 16199 ts_manager.cc:194] Registered new tserver with Master: 368a1795ee5d4258bdd74462416d3934 (127.4.197.129:36813)
I20260929 02:51:18.872776 16199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:51771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:51:18.884524 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:49761/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35221
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:19.315471 16419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:19.315790 16419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:19.315929 16419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:19.323926 16419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:19.324081 16419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:19.324245 16419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:51:19.334547 16419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.197.148:49761/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:51:19.336083 16419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:19.337622 16419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:19.353307 16432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:19.353317 16433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:19.353819 16435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:19.354339 16419 server_base.cc:1034] running on GCE node
I20260929 02:51:19.355016 16419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:19.356012 16419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:19.357226 16419 hybrid_clock.cc:648] HybridClock initialized: now 1790650279357189 us; error 30 us; skew 500 ppm
I20260929 02:51:19.357524 16419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:19 dist-test-slave-t6c1 krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650279, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:19.360627 16419 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:51:19.362699 16419 webserver.cc:460] Webserver started at http://127.4.197.130:41111/ using document root <none> and password file <none>
I20260929 02:51:19.363395 16419 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:19.363505 16419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:19.363786 16419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:51:19 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650279, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:51:19.381983 16419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "d0f3fea3c0a442e6bfbfa01037399127"
format_stamp: "Formatted at 2026-09-29 02:51:19 on dist-test-slave-t6c1"
server_key: "c828543d9cdb72f432f4523acfceb95b"
server_key_iv: "5c366bea9d29dfeefb936ec9dfb1909e"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:19.383625 16419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "d0f3fea3c0a442e6bfbfa01037399127"
format_stamp: "Formatted at 2026-09-29 02:51:19 on dist-test-slave-t6c1"
server_key: "c828543d9cdb72f432f4523acfceb95b"
server_key_iv: "5c366bea9d29dfeefb936ec9dfb1909e"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:19.391026 16419 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260929 02:51:19.408766 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:19.410673 16419 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:51:19.410858 16419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "d0f3fea3c0a442e6bfbfa01037399127"
format_stamp: "Formatted at 2026-09-29 02:51:19 on dist-test-slave-t6c1"
server_key: "c828543d9cdb72f432f4523acfceb95b"
server_key_iv: "5c366bea9d29dfeefb936ec9dfb1909e"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:19.411144 16419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:19.442713 16419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:19.447136 16419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:19.447433 16419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:19.448486 16419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:19.449878 16419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:19.449947 16419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:19.450069 16419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:19.450122 16419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:19.488373 16419 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:46129
I20260929 02:51:19.488436 16554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:46129 every 8 connection(s)
I20260929 02:51:19.489704 16419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:51:19.495227 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 16419
I20260929 02:51:19.495407 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:51:19.495934 4886 external_mini_cluster.cc:1428] Setting key e2027e17b6f158de18de7810e5e49371
Sep 29 02:51:19 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650279, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
I20260929 02:51:19.519681 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:19.492688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:49485 (local address 127.4.197.190:35221)
0929 02:51:19.503273 (+ 10585us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:19.503278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:19.503294 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:51:19.503343 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:19.503346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:19.503406 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:19.503501 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:19.505024 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:19.506156 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:19.508434 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:19.509033 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:19.509148 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:19.513130 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:19.513137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:19.513141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:19.513178 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:19.516439 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:19.517346 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:19.517351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:19.517354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:19.517498 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:19.517967 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:19.517972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:19.517975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:19.518297 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:19.518421 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:19.519107 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 02:51:19.519321 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10482,"thread_start_us":10358,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
I20260929 02:51:19.520624 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:19.493321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:35221 (local address 127.4.197.130:49485)
0929 02:51:19.494243 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:19.494294 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:19.496555 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:19.503819 (+ 7264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:19.503829 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:19.504256 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:19.504847 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:19.504865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:19.506335 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:19.506341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:19.508284 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:19.508298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:19.509259 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:19.509266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:19.509554 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:19.510295 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:19.510322 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:19.512874 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:19.516570 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:19.516578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:19.516595 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:19.517155 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:19.517659 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:19.517662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:19.517665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:19.517848 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:19.518432 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:19.518437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:19.518724 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 02:51:19.519001 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 02:51:19.519414 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":385,"threads_started":1}
I20260929 02:51:19.521998 16555 heartbeater.cc:347] Connected to a master server at 127.4.197.190:35221
I20260929 02:51:19.522313 16555 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:19.523068 16555 heartbeater.cc:511] Master 127.4.197.190:35221 requested a full tablet report, sending...
I20260929 02:51:19.524470 16199 ts_manager.cc:194] Registered new tserver with Master: d0f3fea3c0a442e6bfbfa01037399127 (127.4.197.130:46129)
I20260929 02:51:19.525261 16199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:49485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:51:19.547775 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:49761/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35221
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:51:19.877121 16413 heartbeater.cc:503] Master 127.4.197.190:35221 was elected leader, sending a full tablet report...
W20260929 02:51:19.888840 16561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:19.889158 16561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:19.889293 16561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:19.896939 16561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:19.897059 16561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:19.897207 16561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:51:19.907294 16561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.197.148:49761/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:35221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:51:19.908469 16561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:19.909670 16561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:19.921676 16566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:19.921653 16567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:19.921800 16569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:19.923122 16561 server_base.cc:1034] running on GCE node
I20260929 02:51:19.923687 16561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:19.924549 16561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:19.925745 16561 hybrid_clock.cc:648] HybridClock initialized: now 1790650279925698 us; error 44 us; skew 500 ppm
I20260929 02:51:19.926028 16561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:19 dist-test-slave-t6c1 krb5kdc[15722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650279, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:19.928864 16561 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:51:19.930171 16561 webserver.cc:460] Webserver started at http://127.4.197.131:34671/ using document root <none> and password file <none>
I20260929 02:51:19.930795 16561 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:19.930904 16561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:19.931200 16561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:51:19 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650279, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:51:19.944303 16561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "a90c2b45c2dc44b0a022c849469d8365"
format_stamp: "Formatted at 2026-09-29 02:51:19 on dist-test-slave-t6c1"
server_key: "d023ef2515f8ab2d1cfd125a903d71b4"
server_key_iv: "aa4593fed084633b32160850426621fd"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:19.945148 16561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "a90c2b45c2dc44b0a022c849469d8365"
format_stamp: "Formatted at 2026-09-29 02:51:19 on dist-test-slave-t6c1"
server_key: "d023ef2515f8ab2d1cfd125a903d71b4"
server_key_iv: "aa4593fed084633b32160850426621fd"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:19.950839 16561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:51:19.966452 16576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:19.968015 16561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:51:19.968183 16561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "a90c2b45c2dc44b0a022c849469d8365"
format_stamp: "Formatted at 2026-09-29 02:51:19 on dist-test-slave-t6c1"
server_key: "d023ef2515f8ab2d1cfd125a903d71b4"
server_key_iv: "aa4593fed084633b32160850426621fd"
server_key_version: "kuduclusterkey@0"
I20260929 02:51:19.968400 16561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:19.984783 16561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:19.988741 16561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:19.989024 16561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:19.990394 16561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:19.991938 16561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:19.992022 16561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:19.992141 16561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:19.992192 16561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:20.025947 16561 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:37537
I20260929 02:51:20.025990 16688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:37537 every 8 connection(s)
I20260929 02:51:20.027304 16561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:51:20.032936 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 16561
I20260929 02:51:20.033119 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:51:20.033569 4886 external_mini_cluster.cc:1428] Setting key fa09c50f3fd2810736d73870ba175b9e
Sep 29 02:51:20 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650279, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:51:20.048861 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.030114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:45355 (local address 127.4.197.190:35221)
0929 02:51:20.030584 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.030589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.031693 (+ 1104us) server_negotiation.cc:407] Connection header received
0929 02:51:20.033642 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.033650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.033735 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.033835 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:20.036218 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.037206 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.038990 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.039440 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.039537 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.042910 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:20.042940 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:20.042946 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:20.042988 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:20.045552 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:20.046347 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:20.046352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:20.046354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:20.046501 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:20.046879 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:20.046882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:20.046884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:20.047298 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:20.047475 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.048253 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.048505 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":200,"threads_started":1}
I20260929 02:51:20.050107 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.030749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:35221 (local address 127.4.197.131:45355)
0929 02:51:20.031507 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.031556 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:20.033349 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:20.034114 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:20.034126 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:20.034663 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:20.036030 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.036058 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.037359 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.037363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.038826 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.038846 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.039631 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.039636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.039855 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.040537 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:20.040562 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:20.042685 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:20.045710 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:20.045718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:20.045733 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:20.046190 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:20.046595 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:20.046599 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:20.046601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:20.046787 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:20.047424 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:20.047430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:20.047885 (+ 455us) client_negotiation.cc:769] Sending connection context
0929 02:51:20.048181 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 02:51:20.048702 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":302,"threads_started":1}
I20260929 02:51:20.051376 16689 heartbeater.cc:347] Connected to a master server at 127.4.197.190:35221
I20260929 02:51:20.051666 16689 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:20.052413 16689 heartbeater.cc:511] Master 127.4.197.190:35221 requested a full tablet report, sending...
I20260929 02:51:20.053907 16199 ts_manager.cc:194] Registered new tserver with Master: a90c2b45c2dc44b0a022c849469d8365 (127.4.197.131:37537)
I20260929 02:51:20.054721 16199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:45355
I20260929 02:51:20.062160 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:51:20.079342 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.066552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.4.197.190:35221)
0929 02:51:20.066699 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.066703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.067189 (+ 486us) server_negotiation.cc:407] Connection header received
0929 02:51:20.067384 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.067388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.067485 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.067576 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:20.069392 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.070093 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.071421 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.071988 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.072096 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.074290 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:20.074294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:20.074297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:20.074329 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:20.076727 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:20.077375 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:20.077400 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:20.077403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:20.077539 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:20.077899 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:20.077904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:20.077907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:20.078188 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:20.078419 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.078760 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.079031 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:51:20.083963 16199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:51:20.091516 16199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:51:20.114940 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.105600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.4.197.131:37537)
0929 02:51:20.106492 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.106502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.107084 (+ 582us) server_negotiation.cc:407] Connection header received
0929 02:51:20.107922 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.107930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.108136 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.108302 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:20.108812 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.109840 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.112658 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.113820 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.113934 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.114148 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.114535 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.114654 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":552,"threads_started":1}
I20260929 02:51:20.115285 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.103815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:46129 (local address 127.0.0.1:49398)
0929 02:51:20.105468 (+ 1653us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.105488 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:20.105588 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:20.108010 (+ 2422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:20.108015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:20.108038 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:20.108380 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.108395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.110127 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.110132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.111828 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.111841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.113397 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.113403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.113585 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.113633 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 02:51:20.113744 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:51:20.113937 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"mutex_wait_us":23,"thread_start_us":586,"threads_started":1}
I20260929 02:51:20.115491 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.103995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.4.197.129:36813)
0929 02:51:20.106122 (+ 2127us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.106131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.106160 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:51:20.106254 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.106260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.106526 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.107465 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:20.107943 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.109591 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.112390 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.113740 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.114592 (+ 852us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.114872 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.114975 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.115130 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1954,"thread_start_us":1814,"threads_started":1}
I20260929 02:51:20.115612 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.104300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.4.197.130:46129)
0929 02:51:20.107448 (+ 3148us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.107455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.107480 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:51:20.107565 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.107571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.107773 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.108007 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:20.108610 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.109986 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.111975 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.113185 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.114940 (+ 1755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.115114 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.115172 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.115316 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3014,"thread_start_us":2907,"threads_started":1}
I20260929 02:51:20.115669 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.105147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:37537 (local address 127.0.0.1:43782)
0929 02:51:20.106966 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.106988 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:20.107736 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:20.108321 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:20.108325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:20.108348 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:20.108666 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.108675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.109982 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.109986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.112531 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.112540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.114209 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.114214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.114342 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.114364 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:51:20.115225 (+ 861us) client_negotiation.cc:240] Negotiation successful
0929 02:51:20.115347 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1658,"thread_start_us":1520,"threads_started":1}
I20260929 02:51:20.116173 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.103695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36813 (local address 127.0.0.1:41564)
0929 02:51:20.104761 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.104796 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:20.104953 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:20.106759 (+ 1806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:20.106770 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:20.106801 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:20.107279 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.107292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.109728 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.109733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.112216 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.112230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.113954 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.113960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.114122 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.114156 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:51:20.114224 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 02:51:20.114378 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":83,"thread_start_us":1497,"threads_started":1}
I20260929 02:51:20.119372 16624 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd83cc82bdc4bba969bbf7188b42c50 (DEFAULT_TABLE table=test-table [id=50fafe3152244f25998cdd758c0d2b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:20.119472 16347 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd83cc82bdc4bba969bbf7188b42c50 (DEFAULT_TABLE table=test-table [id=50fafe3152244f25998cdd758c0d2b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:20.119388 16490 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd83cc82bdc4bba969bbf7188b42c50 (DEFAULT_TABLE table=test-table [id=50fafe3152244f25998cdd758c0d2b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:20.120543 16624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd83cc82bdc4bba969bbf7188b42c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:20.120622 16347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd83cc82bdc4bba969bbf7188b42c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:20.120622 16490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd83cc82bdc4bba969bbf7188b42c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:20.131135 16708 tablet_bootstrap.cc:492] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: Bootstrap starting.
I20260929 02:51:20.133977 16708 tablet_bootstrap.cc:654] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:20.134785 16709 tablet_bootstrap.cc:492] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934: Bootstrap starting.
I20260929 02:51:20.135550 16710 tablet_bootstrap.cc:492] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: Bootstrap starting.
I20260929 02:51:20.136081 16708 log.cc:824] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:20.138657 16710 tablet_bootstrap.cc:654] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:20.139079 16708 tablet_bootstrap.cc:492] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: No bootstrap required, opened a new log
I20260929 02:51:20.139276 16709 tablet_bootstrap.cc:654] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:20.139328 16708 ts_tablet_manager.cc:1397] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:51:20.139832 16710 log.cc:824] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:20.140182 16709 log.cc:824] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:20.142699 16710 tablet_bootstrap.cc:492] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: No bootstrap required, opened a new log
I20260929 02:51:20.142781 16709 tablet_bootstrap.cc:492] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934: No bootstrap required, opened a new log
I20260929 02:51:20.142998 16710 ts_tablet_manager.cc:1397] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260929 02:51:20.143059 16709 ts_tablet_manager.cc:1397] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:51:20.147596 16708 raft_consensus.cc:348] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 } } peers { permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 } } peers { permanent_uuid: "a90c2b45c2dc44b0a022c849469d8365" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 37537 } }
I20260929 02:51:20.147938 16708 raft_consensus.cc:374] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:20.148052 16708 raft_consensus.cc:729] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90c2b45c2dc44b0a022c849469d8365, State: Initialized, Role: FOLLOWER
I20260929 02:51:20.148584 16708 consensus_queue.cc:260] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 } } peers { permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 } } peers { permanent_uuid: "a90c2b45c2dc44b0a022c849469d8365" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 37537 } }
I20260929 02:51:20.149575 16689 heartbeater.cc:503] Master 127.4.197.190:35221 was elected leader, sending a full tablet report...
I20260929 02:51:20.150283 16708 ts_tablet_manager.cc:1428] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 02:51:20.153702 16709 raft_consensus.cc:348] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 } } peers { permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 } } peers { permanent_uuid: "a90c2b45c2dc44b0a022c849469d8365" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 37537 } }
I20260929 02:51:20.153931 16710 raft_consensus.cc:348] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 } } peers { permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 } } peers { permanent_uuid: "a90c2b45c2dc44b0a022c849469d8365" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 37537 } }
I20260929 02:51:20.154275 16709 raft_consensus.cc:374] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:20.154356 16710 raft_consensus.cc:374] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:20.154428 16709 raft_consensus.cc:729] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 368a1795ee5d4258bdd74462416d3934, State: Initialized, Role: FOLLOWER
I20260929 02:51:20.154486 16710 raft_consensus.cc:729] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f3fea3c0a442e6bfbfa01037399127, State: Initialized, Role: FOLLOWER
I20260929 02:51:20.155303 16709 consensus_queue.cc:260] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 } } peers { permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 } } peers { permanent_uuid: "a90c2b45c2dc44b0a022c849469d8365" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 37537 } }
I20260929 02:51:20.155305 16710 consensus_queue.cc:260] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 } } peers { permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 } } peers { permanent_uuid: "a90c2b45c2dc44b0a022c849469d8365" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 37537 } }
I20260929 02:51:20.156927 16555 heartbeater.cc:503] Master 127.4.197.190:35221 was elected leader, sending a full tablet report...
I20260929 02:51:20.157341 16709 ts_tablet_manager.cc:1428] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260929 02:51:20.157758 16710 ts_tablet_manager.cc:1428] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260929 02:51:20.241988 16556 tablet.cc:2369] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:20.270267 16714 raft_consensus.cc:482] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:51:20.270560 16714 raft_consensus.cc:504] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 } } peers { permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 } } peers { permanent_uuid: "a90c2b45c2dc44b0a022c849469d8365" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 37537 } }
I20260929 02:51:20.272409 16714 leader_election.cc:288] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 368a1795ee5d4258bdd74462416d3934 (127.4.197.129:36813), d0f3fea3c0a442e6bfbfa01037399127 (127.4.197.130:46129)
W20260929 02:51:20.278879 16690 tablet.cc:2369] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:20.279155 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.272769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:33109 (local address 127.4.197.129:36813)
0929 02:51:20.272987 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.272994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.273016 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:51:20.273145 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.273150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.273238 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.273334 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:20.273826 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.274843 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.276867 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.278131 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.278266 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.278323 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.278672 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.278798 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:51:20.279174 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.272611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36813 (local address 127.4.197.131:33109)
0929 02:51:20.272821 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.272842 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:20.272971 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:20.273461 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:20.273466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:20.273500 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:20.273652 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.273662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.275031 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.275037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.276702 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.276713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.278294 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.278300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.278473 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.278508 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:51:20.278630 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 02:51:20.278763 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":3}
I20260929 02:51:20.280361 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.273117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:57169 (local address 127.4.197.130:46129)
0929 02:51:20.273327 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.273333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.273699 (+ 366us) server_negotiation.cc:407] Connection header received
0929 02:51:20.273796 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.273802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.273874 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.273951 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:20.275012 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.276068 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.278161 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.279332 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.279490 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.279556 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.280004 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.280127 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:51:20.280392 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.272821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:46129 (local address 127.4.197.131:57169)
0929 02:51:20.273587 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.273604 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:20.273700 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:20.273987 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:20.273991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:20.274014 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:20.274840 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.274851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.276209 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.276212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.278013 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:20.278028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.279680 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.279685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:20.279848 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.279870 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:51:20.279948 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:51:20.280069 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":437,"threads_started":1}
I20260929 02:51:20.280256 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd83cc82bdc4bba969bbf7188b42c50" candidate_uuid: "a90c2b45c2dc44b0a022c849469d8365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "368a1795ee5d4258bdd74462416d3934" is_pre_election: true
I20260929 02:51:20.280797 16367 raft_consensus.cc:2454] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a90c2b45c2dc44b0a022c849469d8365 in term 0.
I20260929 02:51:20.281467 16579 leader_election.cc:302] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 368a1795ee5d4258bdd74462416d3934, a90c2b45c2dc44b0a022c849469d8365; no voters:
I20260929 02:51:20.281374 16510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd83cc82bdc4bba969bbf7188b42c50" candidate_uuid: "a90c2b45c2dc44b0a022c849469d8365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f3fea3c0a442e6bfbfa01037399127" is_pre_election: true
I20260929 02:51:20.281817 16714 raft_consensus.cc:2781] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:51:20.281829 16510 raft_consensus.cc:2454] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a90c2b45c2dc44b0a022c849469d8365 in term 0.
I20260929 02:51:20.281924 16714 raft_consensus.cc:482] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:51:20.282045 16714 raft_consensus.cc:3037] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:20.284459 16714 raft_consensus.cc:504] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 } } peers { permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 } } peers { permanent_uuid: "a90c2b45c2dc44b0a022c849469d8365" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 37537 } }
I20260929 02:51:20.285082 16714 leader_election.cc:288] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [CANDIDATE]: Term 1 election: Requested vote from peers 368a1795ee5d4258bdd74462416d3934 (127.4.197.129:36813), d0f3fea3c0a442e6bfbfa01037399127 (127.4.197.130:46129)
I20260929 02:51:20.285571 16510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd83cc82bdc4bba969bbf7188b42c50" candidate_uuid: "a90c2b45c2dc44b0a022c849469d8365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f3fea3c0a442e6bfbfa01037399127"
I20260929 02:51:20.285574 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd83cc82bdc4bba969bbf7188b42c50" candidate_uuid: "a90c2b45c2dc44b0a022c849469d8365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "368a1795ee5d4258bdd74462416d3934"
I20260929 02:51:20.285745 16510 raft_consensus.cc:3037] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:20.285763 16367 raft_consensus.cc:3037] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:20.288098 16367 raft_consensus.cc:2454] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a90c2b45c2dc44b0a022c849469d8365 in term 1.
I20260929 02:51:20.288388 16510 raft_consensus.cc:2454] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a90c2b45c2dc44b0a022c849469d8365 in term 1.
I20260929 02:51:20.288564 16579 leader_election.cc:302] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 368a1795ee5d4258bdd74462416d3934, a90c2b45c2dc44b0a022c849469d8365; no voters:
I20260929 02:51:20.288877 16714 raft_consensus.cc:2781] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:20.289288 16714 raft_consensus.cc:686] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 1 LEADER]: Becoming Leader. State: Replica: a90c2b45c2dc44b0a022c849469d8365, State: Running, Role: LEADER
I20260929 02:51:20.289793 16714 consensus_queue.cc:237] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 } } peers { permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 } } peers { permanent_uuid: "a90c2b45c2dc44b0a022c849469d8365" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 37537 } }
I20260929 02:51:20.294879 16198 catalog_manager.cc:5495] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 reported cstate change: term changed from 0 to 1, leader changed from <none> to a90c2b45c2dc44b0a022c849469d8365 (127.4.197.131). New cstate: current_term: 1 leader_uuid: "a90c2b45c2dc44b0a022c849469d8365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a90c2b45c2dc44b0a022c849469d8365" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 37537 } health_report { overall_health: HEALTHY } } }
W20260929 02:51:20.316859 16414 tablet.cc:2369] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:20.354342 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.348855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.4.197.131:37537)
0929 02:51:20.349049 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.349054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.349074 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:51:20.349313 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.349316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.349393 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.349524 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:20.349898 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.350607 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.352174 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.352568 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.352678 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.353251 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:20.353750 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:20.353894 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.354003 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.354085 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:51:20.363366 16367 raft_consensus.cc:1261] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934 [term 1 FOLLOWER]: Refusing update from remote peer a90c2b45c2dc44b0a022c849469d8365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:20.363355 16510 raft_consensus.cc:1261] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127 [term 1 FOLLOWER]: Refusing update from remote peer a90c2b45c2dc44b0a022c849469d8365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:20.364365 16718 consensus_queue.cc:1035] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f3fea3c0a442e6bfbfa01037399127" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:20.364734 16714 consensus_queue.cc:1035] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "368a1795ee5d4258bdd74462416d3934" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 36813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 02:51:20.380128 16721 mvcc.cc:205] Tried to move back new op lower bound from 7334503548357865472 to 7334503548071059456. Current Snapshot: MvccSnapshot[applied={T|T < 7334503548357865472 or (T in {7334503548357865472})}]
I20260929 02:51:20.392843 16198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:51:20.393137 16198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:51:20.394551 16722 mvcc.cc:205] Tried to move back new op lower bound from 7334503548357865472 to 7334503548071059456. Current Snapshot: MvccSnapshot[applied={T|T < 7334503548357865472 or (T in {7334503548357865472})}]
I20260929 02:51:20.398465 16198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da2188619870417fb1863680eebf2b37: Sending DeleteTablet for 3 replicas of tablet 0fd83cc82bdc4bba969bbf7188b42c50
I20260929 02:51:20.399742 16490 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd83cc82bdc4bba969bbf7188b42c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:20 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:49398
I20260929 02:51:20.399953 16624 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd83cc82bdc4bba969bbf7188b42c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:20 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:43782
I20260929 02:51:20.402052 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 16279
I20260929 02:51:20.403021 16347 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd83cc82bdc4bba969bbf7188b42c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:20 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:41564
I20260929 02:51:20.413694 16731 tablet_replica.cc:330] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: stopping tablet replica
I20260929 02:51:20.414204 16731 raft_consensus.cc:2229] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:51:20.415038 16731 raft_consensus.cc:2258] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:20.416100 16732 tablet_replica.cc:330] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: stopping tablet replica
I20260929 02:51:20.416575 16732 raft_consensus.cc:2229] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:51:20.417181 16732 raft_consensus.cc:2258] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 02:51:20 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:51:20.418177 16731 ts_tablet_manager.cc:1905] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:20.418844 16732 ts_tablet_manager.cc:1905] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:20.425434 16732 ts_tablet_manager.cc:1918] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:20.425639 16732 log.cc:1196] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/0fd83cc82bdc4bba969bbf7188b42c50
I20260929 02:51:20.426069 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.405320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.4.197.129:36813)
0929 02:51:20.405489 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.405494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.407841 (+ 2347us) server_negotiation.cc:407] Connection header received
0929 02:51:20.408027 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.408033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.408113 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.408211 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:20.409835 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.410958 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.412907 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.413539 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.413663 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.418894 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:20.418938 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:20.418956 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:20.419007 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:20.422851 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:20.423848 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:20.423857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:20.423860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:20.424040 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:20.424415 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:20.424420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:20.424423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:20.424863 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:20.425167 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.425466 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.425681 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:51:20.426172 16732 ts_tablet_manager.cc:1939] T 0fd83cc82bdc4bba969bbf7188b42c50 P a90c2b45c2dc44b0a022c849469d8365: Deleting consensus metadata
I20260929 02:51:20.427017 16407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:20.427960 16183 catalog_manager.cc:4841] TS a90c2b45c2dc44b0a022c849469d8365 (127.4.197.131:37537): tablet 0fd83cc82bdc4bba969bbf7188b42c50 (table test-table [id=50fafe3152244f25998cdd758c0d2b9d]) successfully deleted
I20260929 02:51:20.430100 16731 ts_tablet_manager.cc:1918] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:20.430238 16731 log.cc:1196] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/0fd83cc82bdc4bba969bbf7188b42c50
I20260929 02:51:20.430611 16731 ts_tablet_manager.cc:1939] T 0fd83cc82bdc4bba969bbf7188b42c50 P d0f3fea3c0a442e6bfbfa01037399127: Deleting consensus metadata
I20260929 02:51:20.432324 16183 catalog_manager.cc:4841] TS d0f3fea3c0a442e6bfbfa01037399127 (127.4.197.130:46129): tablet 0fd83cc82bdc4bba969bbf7188b42c50 (table test-table [id=50fafe3152244f25998cdd758c0d2b9d]) successfully deleted
I20260929 02:51:20.536087 16733 tablet_replica.cc:330] T 0fd83cc82bdc4bba969bbf7188b42c50 P 368a1795ee5d4258bdd74462416d3934: stopping tablet replica
I20260929 02:51:20.536314 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 16279
I20260929 02:51:20.550717 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 16419
W20260929 02:51:20.550836 16185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.197.129:36813 (error 108)
I20260929 02:51:20.553035 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.552528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36813 (local address 127.0.0.1:41584)
0929 02:51:20.552745 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.552858 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 02:51:20.553287 16185 catalog_manager.cc:4584] TS 368a1795ee5d4258bdd74462416d3934 (127.4.197.129:36813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fd83cc82bdc4bba969bbf7188b42c50: Network error: Client connection negotiation failed: client connection to 127.4.197.129:36813: connect: Connection refused (error 111)
Sep 29 02:51:20 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:51:20.567783 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.551596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.4.197.130:46129)
0929 02:51:20.552206 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.552212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.552230 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:51:20.552294 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.552299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.552392 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.552500 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:20.554102 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.554871 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.556165 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.556562 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.556672 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.560989 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:20.561008 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:20.561027 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:20.561069 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:20.564806 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:20.565574 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:20.565583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:20.565587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:20.565739 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:20.566159 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:20.566164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:20.566168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:20.566614 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:20.566910 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.567201 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.567417 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486}
I20260929 02:51:20.568513 16550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:20.607291 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.606699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36813 (local address 127.0.0.1:41590)
0929 02:51:20.606967 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.607101 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:51:20.608415 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.608017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36813 (local address 127.0.0.1:41604)
0929 02:51:20.608171 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.608258 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:51:20.658241 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 16419
I20260929 02:51:20.669103 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.668487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36813 (local address 127.0.0.1:41610)
0929 02:51:20.668759 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.668900 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 02:51:20.670360 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.669936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36813 (local address 127.0.0.1:41614)
0929 02:51:20.670109 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.670206 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 02:51:20.672808 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 16561
Sep 29 02:51:20 dist-test-slave-t6c1 krb5kdc[15722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:51:20.689951 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.673814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.4.197.131:37537)
0929 02:51:20.674017 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:20.674023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:20.674047 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:51:20.674117 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:20.674123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:20.674193 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:20.674278 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:20.675734 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.676606 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.677979 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:20.678486 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:20.678606 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:20.683662 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:20.683674 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:20.683689 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:20.683754 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:20.686810 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:20.687770 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:20.687778 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:20.687780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:20.687982 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:20.688450 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:20.688453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:20.688455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:20.688818 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:20.689112 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:20.689425 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 02:51:20.689598 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:51:20.690696 16684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:20.751344 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.750710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36813 (local address 127.0.0.1:41630)
0929 02:51:20.751041 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.751149 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 02:51:20.752481 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.752071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36813 (local address 127.0.0.1:41642)
0929 02:51:20.752283 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.752363 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 02:51:20.808748 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 16561
I20260929 02:51:20.823025 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 16167
I20260929 02:51:20.824275 16229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:20.934994 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:20.934131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:36813 (local address 127.0.0.1:41644)
0929 02:51:20.934653 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:20.934770 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386}
I20260929 02:51:20.935266 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 16167
I20260929 02:51:20.951438 4886 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 02:51:21.287222 4886 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 02:51:21.287436 4886 mini_ranger.cc:66] Stopping Ranger...
I20260929 02:51:21.623284 4886 mini_ranger.cc:68] Stopped Ranger
2026-09-29 02:51:21.623 UTC [15752] LOG: received smart shutdown request
2026-09-29 02:51:21.626 UTC [15752] LOG: background worker "logical replication launcher" (PID 15761) exited with exit code 1
2026-09-29 02:51:21.627 UTC [15756] LOG: shutting down
2026-09-29 02:51:21.800 UTC [15752] LOG: database system is shut down
2026-09-29T02:51:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49222 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:22 dist-test-slave-t6c1 krb5kdc[16748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:22 dist-test-slave-t6c1 krb5kdc[16748](info): set up 2 sockets
Sep 29 02:51:22 dist-test-slave-t6c1 krb5kdc[16748](info): commencing operation
krb5kdc: starting...
W20260929 02:51:22.931187 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:51:23 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.197.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.197.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/rangeradmin_127.4.197.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/rangeradmin_127.4.197.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.197.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.197.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/rangerlookup_127.4.197.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/rangerlookup_127.4.197.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.197.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.197.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/HTTP_127.4.197.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/HTTP_127.4.197.148@KRBTEST.COM.keytab.
I20260929 02:51:23.234269 4886 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgLoZGx/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/postgres -l logfile start
2026-09-29 02:51:25.670 UTC [16778] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 02:51:25.670 UTC [16778] LOG: listening on IPv4 address "127.4.197.148", port 41039
2026-09-29 02:51:25.673 UTC [16778] LOG: listening on Unix socket "/tmp/.s.PGSQL.41039"
2026-09-29 02:51:25.692 UTC [16781] LOG: database system was shut down at 2026-09-29 02:51:24 UTC
2026-09-29 02:51:25.695 UTC [16778] LOG: database system is ready to accept connections
I20260929 02:51:26.578789 4886 mini_postgres.cc:96] Postgres bound to 41039
2026-09-29 02:51:26.594 UTC [16789] FATAL: database "slave" does not exist
127.4.197.148:41039 - accepting connections
I20260929 02:51:26.594651 4886 mini_ranger.cc:161] Starting Ranger...
I20260929 02:51:26.618636 4886 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 02:51:27.081960 4886 mini_ranger.cc:87] Created ranger Postgres database
I20260929 02:51:27.082053 4886 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-admin
2026-09-29 02:51:27,368 [I] DB FLAVOR :POSTGRES
2026-09-29 02:51:27,369 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:51:27,369 [I] Checking connection..
2026-09-29 02:51:27,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 02:51:27,611 [I] Checking connection passed.
2026-09-29 02:51:27,611 [I] --------- Verifying version history table ---------
2026-09-29 02:51:27,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:51:27,850 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:51:27,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:51:28,105 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:51:28,105 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 02:51:28,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 02:51:28.351 UTC [16853] WARNING: there is no transaction in progress
2026-09-29 02:51:28,357 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 02:51:28,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:51:28,593 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 02:51:28,593 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:51:28,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 02:51:28,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 02:51:29,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t6c1.c.gcp-upstream.internal','N') ;"
2026-09-29 02:51:29,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:51:29,579 [I] Table x_portal_user does not exist in database ranger
2026-09-29 02:51:29,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:51:29,828 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 02:51:29,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:51:30,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:51:30,325 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 02:51:30,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 02:51:31.296 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.311 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.326 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.341 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.357 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.376 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.439 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.446 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.461 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.470 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.481 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.495 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.504 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.511 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.518 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:31.525 UTC [16961] WARNING: there is no transaction in progress
2026-09-29 02:51:32,148 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 02:51:32,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 02:51:32,387 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 02:51:32,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:51:32,647 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 02:51:32,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:51:32,901 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 02:51:32,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:51:33,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:51:33,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t6c1.c.gcp-upstream.internal';"
2026-09-29 02:51:33,649 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 02:51:33,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 02:51:33,892 [I] DB_PATCHES have already been applied
I20260929 02:51:33.897970 4886 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgLoZGx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 02:51:33.898053 4886 mini_ranger.cc:193] Using host: 127.4.197.148
I20260929 02:51:33.910583 4886 mini_ranger.cc:227] Ranger admin URL: http://127.4.197.148:33579
Sep 29, 2026 2:51:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:51:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:51:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:51:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:51:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:51:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:51:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 02:51:34.931748 4886 mini_ranger.cc:160] Time spent starting Ranger: real 8.337s user 0.003s sys 0.033s
Sep 29, 2026 2:51:53 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:51:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 2:51:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:51:53 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:51:54 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 02:51:55.431535 4886 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.197.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.197.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/rangerkms_127.4.197.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.197.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/rangerkms_127.4.197.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.197.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.197.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.197.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/HTTP_127.4.197.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.197.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/HTTP_127.4.197.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 02:51:55 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650315, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 02:51:55.600715 4886 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 02:51:55.621596 4886 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 02:51:56.405534 4886 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 02:51:56.405661 4886 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-kms
I20260929 02:51:56.405707 4886 mini_ranger_kms.cc:227] Using postgres at 127.4.197.148:41039
2026-09-29 02:51:56,746 [I] DB FLAVOR :POSTGRES
2026-09-29 02:51:56,747 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:51:56,747 [I] Checking connection
2026-09-29 02:51:56,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 02:51:56,975 [I] connection success
2026-09-29 02:51:56,975 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 02:51:56,975 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 02:51:56,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 02:51:57,232 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 02:51:57,233 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:51:57,233 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 02:51:57,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.197.148:41039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 02:51:57,478 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 02:51:57.958544 4886 mini_ranger_kms.cc:310] Created kms service
I20260929 02:51:58.182860 4886 mini_ranger_kms.cc:326] Created kudu user
I20260929 02:51:58.223718 4886 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 02:51:58 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650315, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:51:58.580443 4886 mini_ranger_kms.cc:384] Added ranger policy
I20260929 02:51:58.580768 4886 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgLoZGx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgLoZGx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 02:51:58.580824 4886 mini_ranger_kms.cc:240] Using host: 127.4.197.148
I20260929 02:51:58.598366 4886 mini_ranger_kms.cc:276] Ranger KMS PID: 17141
I20260929 02:51:58.598495 4886 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.197.148:51961
Sep 29, 2026 2:51:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:51:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:51:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:51:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:51:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:51:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 02:52:00.629393 4886 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.029s user 0.070s sys 0.289s
I20260929 02:52:00.629637 4886 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 02:52:00.631166 4886 mini_ranger_kms.cc:401] 127.4.197.148:51961/kms/v1/keys
Sep 29 02:52:00 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650320, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.197.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:52:01 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650320, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.197.148@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
Sep 29 02:52:01 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650320, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.197.148@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
Sep 29 02:52:03 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:52:03.955423 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33121
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:33121
--ranger_config_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:51961/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:04.274950 17192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:04.275249 17192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:04.275347 17192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:04.282464 17192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:52:04.282593 17192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:04.282644 17192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:04.282701 17192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:52:04.282747 17192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:52:04.292848 17192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.197.148:51961/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:33121
--ranger_config_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:04.294049 17192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:04.295409 17192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:04.307399 17197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:04.307432 17200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:04.307468 17198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:04.308001 17192 server_base.cc:1034] running on GCE node
I20260929 02:52:04.308836 17192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:04.310271 17192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:04.311498 17192 hybrid_clock.cc:648] HybridClock initialized: now 1790650324311440 us; error 36 us; skew 500 ppm
I20260929 02:52:04.311860 17192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:04 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650324, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:04.314873 17192 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:52:04.316471 17192 webserver.cc:460] Webserver started at http://127.4.197.190:34865/ using document root <none> and password file <none>
I20260929 02:52:04.317209 17192 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:04.317318 17192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:04.317612 17192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:52:04 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650324, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:52:04.376381 17192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "eac360019a69471eac3e11b6463e709c"
format_stamp: "Formatted at 2026-09-29 02:52:04 on dist-test-slave-t6c1"
server_key: "28c962949dbc734220e9d22ba42a21d4"
server_key_iv: "cffb58c2f99b8891d154a68509253b8c"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:04.377329 17192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "eac360019a69471eac3e11b6463e709c"
format_stamp: "Formatted at 2026-09-29 02:52:04 on dist-test-slave-t6c1"
server_key: "28c962949dbc734220e9d22ba42a21d4"
server_key_iv: "cffb58c2f99b8891d154a68509253b8c"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:04.382860 17192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:52:04.403558 17207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:04.405283 17192 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260929 02:52:04.405465 17192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "eac360019a69471eac3e11b6463e709c"
format_stamp: "Formatted at 2026-09-29 02:52:04 on dist-test-slave-t6c1"
server_key: "28c962949dbc734220e9d22ba42a21d4"
server_key_iv: "cffb58c2f99b8891d154a68509253b8c"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:04.405721 17192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:04.440058 17192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:04.444321 17192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:04.444619 17192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:04.461247 17192 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:33121
I20260929 02:52:04.461266 17258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:33121 every 8 connection(s)
I20260929 02:52:04.462568 17192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:52:04.466446 17259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:04.468232 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 17192
I20260929 02:52:04.468400 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:52:04.469031 4886 external_mini_cluster.cc:1428] Setting key 02e348beb79659680ac3f8018e000bfe
I20260929 02:52:04.478652 17259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: Bootstrap starting.
I20260929 02:52:04.482875 17259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: Neither blocks nor log segments found. Creating new log.
Sep 29 02:52:04 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:04.484297 17259 log.cc:824] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:04.487846 17259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: No bootstrap required, opened a new log
I20260929 02:52:04.492504 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:04.470859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.4.197.190:33121)
0929 02:52:04.471665 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:04.471677 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:04.471719 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:52:04.473392 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:04.473414 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:04.473812 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:04.474158 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:04.475945 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:04.477345 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:04.479229 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:04.479942 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:04.480103 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:04.485271 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:04.485285 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:04.485313 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:04.485361 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:04.488426 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:04.489266 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:04.489272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:04.489278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:04.489432 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:04.489841 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:04.489844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:04.489846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:04.490385 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:04.490625 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:04.491103 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 02:52:04.491462 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":348,"threads_started":1}
I20260929 02:52:04.496548 17259 raft_consensus.cc:348] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } }
I20260929 02:52:04.496933 17259 raft_consensus.cc:374] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:04.497085 17259 raft_consensus.cc:729] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac360019a69471eac3e11b6463e709c, State: Initialized, Role: FOLLOWER
I20260929 02:52:04.497653 17259 consensus_queue.cc:260] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } }
I20260929 02:52:04.497888 17259 raft_consensus.cc:388] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:52:04.498032 17259 raft_consensus.cc:482] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:52:04.498221 17259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:04.500501 17259 raft_consensus.cc:504] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } }
I20260929 02:52:04.501118 17259 leader_election.cc:302] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eac360019a69471eac3e11b6463e709c; no voters:
I20260929 02:52:04.501709 17259 leader_election.cc:288] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:52:04.501814 17264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:52:04.502446 17264 raft_consensus.cc:686] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 1 LEADER]: Becoming Leader. State: Replica: eac360019a69471eac3e11b6463e709c, State: Running, Role: LEADER
I20260929 02:52:04.502961 17264 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } }
I20260929 02:52:04.503778 17259 sys_catalog.cc:564] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:52:04.505503 17266 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [sys.catalog]: SysCatalogTable state changed. Reason: New leader eac360019a69471eac3e11b6463e709c. Latest consensus state: current_term: 1 leader_uuid: "eac360019a69471eac3e11b6463e709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } } }
I20260929 02:52:04.505915 17266 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:04.511436 17265 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eac360019a69471eac3e11b6463e709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } } }
I20260929 02:52:04.511893 17265 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:04.512194 17272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:52:04.517796 17272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:52:04.518591 17259 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:52:04.537669 17272 catalog_manager.cc:1347] Generated new cluster ID: 75ada9262f444e5f84f690f5b5cf4fb9
I20260929 02:52:04.537847 17272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:52:04.554224 17272 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:52:04.557884 17272 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:52:04.571743 17272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: Generated new TSK 0
I20260929 02:52:04.573093 17272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:52:05.282326 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:51961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33121
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:52:05 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650325, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 02:52:05.686080 17304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:05.686578 17304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:05.686755 17304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:05.695390 17304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:05.695518 17304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:05.695703 17304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:52:05.706892 17304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.197.148:51961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:05.708631 17304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:05.709957 17304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:05.725258 17314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:05.726008 17315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:05.740459 17304 server_base.cc:1034] running on GCE node
W20260929 02:52:05.740746 17317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:05.741478 17304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:05.742883 17304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:05.744237 17304 hybrid_clock.cc:648] HybridClock initialized: now 1790650325744163 us; error 72 us; skew 500 ppm
I20260929 02:52:05.744674 17304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:05 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650325, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:05.753194 17304 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:52:05.755563 17304 webserver.cc:460] Webserver started at http://127.4.197.129:42601/ using document root <none> and password file <none>
I20260929 02:52:05.756558 17304 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:05.756700 17304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:05.757059 17304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:52:05 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650325, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:52:05.780205 17304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "f72312b67598410a87ec34454e4ca4eb"
format_stamp: "Formatted at 2026-09-29 02:52:05 on dist-test-slave-t6c1"
server_key: "f72f5f598cacdb216c864020302c7fc9"
server_key_iv: "e29299bce0de7c85557fbe79ebadd566"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:05.781404 17304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "f72312b67598410a87ec34454e4ca4eb"
format_stamp: "Formatted at 2026-09-29 02:52:05 on dist-test-slave-t6c1"
server_key: "f72f5f598cacdb216c864020302c7fc9"
server_key_iv: "e29299bce0de7c85557fbe79ebadd566"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:05.788224 17304 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 02:52:05.815670 17324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:05.817548 17304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 02:52:05.817824 17304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "f72312b67598410a87ec34454e4ca4eb"
format_stamp: "Formatted at 2026-09-29 02:52:05 on dist-test-slave-t6c1"
server_key: "f72f5f598cacdb216c864020302c7fc9"
server_key_iv: "e29299bce0de7c85557fbe79ebadd566"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:05.818178 17304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:05.844470 17304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:05.850081 17304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:05.850517 17304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:05.857400 17304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:05.862756 17304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:05.862895 17304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:05.863083 17304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:05.863152 17304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:05.918155 17304 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:40725
I20260929 02:52:05.918305 17436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:40725 every 8 connection(s)
I20260929 02:52:05.919991 17304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:52:05.930505 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 17304
I20260929 02:52:05.930738 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:52:05.931545 4886 external_mini_cluster.cc:1428] Setting key dd057573a686f10b46ac6a0a1a0655e3
Sep 29 02:52:05 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650325, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
I20260929 02:52:05.970227 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:05.924469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:34721 (local address 127.4.197.190:33121)
0929 02:52:05.945272 (+ 20803us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:05.945279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:05.946987 (+ 1708us) server_negotiation.cc:407] Connection header received
0929 02:52:05.949171 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:05.949177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:05.949302 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:05.949402 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:05.952469 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:05.953608 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:05.956379 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:05.957057 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:05.957185 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:05.962035 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:05.962044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:05.962049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:05.962111 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:05.965966 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:05.967105 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:05.967112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:05.967115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:05.967290 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:05.967868 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:05.967874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:05.967877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:05.968265 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:05.968406 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:05.969558 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0929 02:52:05.969814 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20654,"thread_start_us":20513,"threads_started":1}
I20260929 02:52:05.972908 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:05.923758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33121 (local address 127.4.197.129:34721)
0929 02:52:05.945902 (+ 22144us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:05.945975 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:05.948653 (+ 2678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:05.950428 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:05.950444 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:05.951223 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:05.952232 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:05.952262 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:05.953835 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:05.953841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:05.956148 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:05.956169 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:05.957571 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:05.957580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:05.957899 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:05.958772 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:05.958805 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:05.961699 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:05.966198 (+ 4499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:05.966209 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:05.966230 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:05.966896 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:05.967451 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:05.967456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:05.967459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:05.967728 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:05.968564 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:05.968576 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:05.969066 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 02:52:05.970811 (+ 1745us) client_negotiation.cc:240] Negotiation successful
0929 02:52:05.971430 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21785,"thread_start_us":27307,"threads_started":1}
I20260929 02:52:05.975347 17437 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33121
I20260929 02:52:05.975690 17437 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:05.976589 17437 heartbeater.cc:511] Master 127.4.197.190:33121 requested a full tablet report, sending...
I20260929 02:52:05.979405 17224 ts_manager.cc:194] Registered new tserver with Master: f72312b67598410a87ec34454e4ca4eb (127.4.197.129:40725)
I20260929 02:52:05.981572 17224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:34721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:52:05.998731 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:51961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33121
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:52:06 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650325, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
Sep 29 02:52:06 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650325, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
W20260929 02:52:06.462484 17443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:06.463038 17443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:06.463245 17443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:06.474839 17443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:06.474998 17443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:06.475251 17443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:52:06.491317 17443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.197.148:51961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:06.493309 17443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:06.495258 17443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:06.533155 17451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:06.534577 17450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:06.538341 17453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:06.538681 17443 server_base.cc:1034] running on GCE node
I20260929 02:52:06.539525 17443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:06.547122 17443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:06.548498 17443 hybrid_clock.cc:648] HybridClock initialized: now 1790650326548423 us; error 71 us; skew 500 ppm
I20260929 02:52:06.548957 17443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:06 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650326, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:06.553282 17443 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:52:06.569024 17443 webserver.cc:460] Webserver started at http://127.4.197.130:37645/ using document root <none> and password file <none>
I20260929 02:52:06.570312 17443 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:06.570557 17443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:06.571044 17443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:52:06 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650326, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:52:06.607995 17443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "9db8ca9195e9443e9e89fddc88f08c09"
format_stamp: "Formatted at 2026-09-29 02:52:06 on dist-test-slave-t6c1"
server_key: "e3807436f850217027c2112c701d843f"
server_key_iv: "898206ae630cb7694611105077043d19"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:06.609405 17443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "9db8ca9195e9443e9e89fddc88f08c09"
format_stamp: "Formatted at 2026-09-29 02:52:06 on dist-test-slave-t6c1"
server_key: "e3807436f850217027c2112c701d843f"
server_key_iv: "898206ae630cb7694611105077043d19"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:06.616257 17443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 02:52:06.648576 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:06.650729 17443 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.010s sys 0.003s
I20260929 02:52:06.651065 17443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "9db8ca9195e9443e9e89fddc88f08c09"
format_stamp: "Formatted at 2026-09-29 02:52:06 on dist-test-slave-t6c1"
server_key: "e3807436f850217027c2112c701d843f"
server_key_iv: "898206ae630cb7694611105077043d19"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:06.651449 17443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:06.675854 17443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:06.679893 17443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:06.680189 17443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:06.681159 17443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:06.682548 17443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:06.682621 17443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:06.682730 17443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:06.682802 17443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:06.712162 17443 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:33609
I20260929 02:52:06.712227 17579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:33609 every 8 connection(s)
I20260929 02:52:06.713351 17443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:52:06.722404 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 17443
I20260929 02:52:06.722545 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:52:06.723147 4886 external_mini_cluster.cc:1428] Setting key c9aa5e1cd27a0b5a0de83b065a37ae15
Sep 29 02:52:06 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650326, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:06.732589 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:06.715688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:46945 (local address 127.4.197.190:33121)
0929 02:52:06.716116 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:06.716121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:06.717059 (+ 938us) server_negotiation.cc:407] Connection header received
0929 02:52:06.718711 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:06.718716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:06.718791 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:06.718892 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:06.720370 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:06.721122 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:06.722581 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:06.723202 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:06.723329 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:06.726610 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:06.726616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:06.726621 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:06.726660 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:06.729734 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:06.730383 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:06.730387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:06.730391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:06.730527 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:06.730874 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:06.730879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:06.730881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:06.731242 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:06.731374 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:06.731982 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 02:52:06.732220 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":222,"threads_started":1}
I20260929 02:52:06.733565 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:06.716272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33121 (local address 127.4.197.130:46945)
0929 02:52:06.716918 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:06.716954 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:06.718473 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:06.719113 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:06.719121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:06.719529 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:06.720184 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:06.720211 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:06.721246 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:06.721257 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:06.722439 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:06.722450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:06.723410 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:06.723416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:06.723615 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:06.724345 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:06.724370 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:06.726397 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:06.729857 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:06.729864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:06.729877 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:06.730285 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:06.730624 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:06.730627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:06.730629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:06.730790 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:06.731343 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:06.731348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:06.731655 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 02:52:06.731875 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 02:52:06.732240 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":256,"threads_started":1}
I20260929 02:52:06.735198 17580 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33121
I20260929 02:52:06.735494 17580 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:06.736181 17580 heartbeater.cc:511] Master 127.4.197.190:33121 requested a full tablet report, sending...
I20260929 02:52:06.737584 17224 ts_manager.cc:194] Registered new tserver with Master: 9db8ca9195e9443e9e89fddc88f08c09 (127.4.197.130:33609)
I20260929 02:52:06.738379 17224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:46945
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:52:06.770216 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:51961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33121
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:52:06.985986 17437 heartbeater.cc:503] Master 127.4.197.190:33121 was elected leader, sending a full tablet report...
W20260929 02:52:07.088356 17586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:07.088630 17586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:07.088748 17586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:07.095996 17586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:07.096112 17586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:07.096274 17586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:52:07.106421 17586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.197.148:51961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:07.107805 17586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:07.109124 17586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:07.121830 17591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:07.121798 17592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:07.121781 17594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:07.122623 17586 server_base.cc:1034] running on GCE node
I20260929 02:52:07.123251 17586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:07.124209 17586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:07.125403 17586 hybrid_clock.cc:648] HybridClock initialized: now 1790650327125349 us; error 47 us; skew 500 ppm
I20260929 02:52:07.125705 17586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:07 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650327, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:07.128756 17586 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:52:07.130183 17586 webserver.cc:460] Webserver started at http://127.4.197.131:37687/ using document root <none> and password file <none>
I20260929 02:52:07.130800 17586 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:07.130892 17586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:07.131184 17586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:52:07 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650327, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:52:07.145478 17586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "e68a3d0c21984595a7013f011d9710e0"
format_stamp: "Formatted at 2026-09-29 02:52:07 on dist-test-slave-t6c1"
server_key: "e5d68be5add73b29729b974cdb7eec80"
server_key_iv: "49fac151d29f9294521cc42658125306"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:07.146384 17586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "e68a3d0c21984595a7013f011d9710e0"
format_stamp: "Formatted at 2026-09-29 02:52:07 on dist-test-slave-t6c1"
server_key: "e5d68be5add73b29729b974cdb7eec80"
server_key_iv: "49fac151d29f9294521cc42658125306"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:07.151885 17586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:52:07.166409 17601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:07.168071 17586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:52:07.168264 17586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "e68a3d0c21984595a7013f011d9710e0"
format_stamp: "Formatted at 2026-09-29 02:52:07 on dist-test-slave-t6c1"
server_key: "e5d68be5add73b29729b974cdb7eec80"
server_key_iv: "49fac151d29f9294521cc42658125306"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:07.168488 17586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:07.197064 17586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:07.201253 17586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:07.201612 17586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:07.202625 17586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:07.204195 17586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:07.204279 17586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:07.204409 17586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:07.204468 17586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:07.236055 17586 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:39509
I20260929 02:52:07.236068 17713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:39509 every 8 connection(s)
I20260929 02:52:07.237402 17586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:52:07.240971 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 17586
I20260929 02:52:07.241153 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:52:07.241804 4886 external_mini_cluster.cc:1428] Setting key cffca1cf87fd110358b1bd66f154c6aa
Sep 29 02:52:07 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650327, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:07.257673 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:07.240021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:42131 (local address 127.4.197.190:33121)
0929 02:52:07.240549 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:07.240557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:07.241650 (+ 1093us) server_negotiation.cc:407] Connection header received
0929 02:52:07.243675 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:07.243681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:07.243783 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:07.243887 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:07.245448 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:07.246241 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:07.247835 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:07.248304 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:07.248410 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:07.251708 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:07.251714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:07.251717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:07.251753 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:07.254296 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:07.254998 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:07.255002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:07.255004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:07.255130 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:07.255633 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:07.255637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:07.255640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:07.255981 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:07.256142 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:07.256959 (+ 817us) server_negotiation.cc:299] Negotiation successful
0929 02:52:07.257215 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":234,"threads_started":1}
I20260929 02:52:07.258733 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:07.240601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33121 (local address 127.4.197.131:42131)
0929 02:52:07.241458 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:07.241508 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:07.243366 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:07.244129 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:07.244138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:07.244544 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:07.245292 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:07.245312 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:07.246391 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:07.246395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:07.247702 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:07.247713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:07.248519 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:07.248524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:07.248732 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:07.249346 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:07.249367 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:07.251455 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:07.254407 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:07.254413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:07.254426 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:07.254870 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:07.255255 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:07.255259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:07.255262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:07.255524 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:07.256099 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:07.256105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:07.256544 (+ 439us) client_negotiation.cc:769] Sending connection context
0929 02:52:07.256850 (+ 306us) client_negotiation.cc:240] Negotiation successful
0929 02:52:07.257544 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":308,"threads_started":1}
I20260929 02:52:07.260118 17714 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33121
I20260929 02:52:07.260411 17714 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:07.261062 17714 heartbeater.cc:511] Master 127.4.197.190:33121 requested a full tablet report, sending...
I20260929 02:52:07.262373 17224 ts_manager.cc:194] Registered new tserver with Master: e68a3d0c21984595a7013f011d9710e0 (127.4.197.131:39509)
I20260929 02:52:07.263217 17224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:42131
I20260929 02:52:07.269881 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:52:07.271649 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 17304
Sep 29 02:52:07 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:52:07.287928 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:07.272439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.4.197.129:40725)
0929 02:52:07.272990 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:07.272999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:07.273028 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:52:07.273215 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:07.273223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:07.273426 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:07.273566 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:07.274997 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:07.275997 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:07.277420 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:07.277964 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:07.278077 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:07.282133 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:07.282145 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:07.282167 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:07.282202 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:07.285180 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:07.285875 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:07.285884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:07.285888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:07.286023 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:07.286359 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:07.286364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:07.286368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:07.286706 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:07.287010 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:07.287335 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 02:52:07.287586 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":239,"threads_started":1}
I20260929 02:52:07.289222 17432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:07.370218 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 17304
I20260929 02:52:07.383011 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 17443
Sep 29 02:52:07 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:52:07.398571 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:07.383818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.4.197.130:33609)
0929 02:52:07.384367 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:07.384374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:07.384399 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:52:07.384458 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:07.384464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:07.384668 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:07.384813 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:07.385914 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:07.386879 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:07.388183 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:07.388721 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:07.388814 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:07.392974 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:07.392986 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:07.393000 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:07.393039 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:07.395921 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:07.396568 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:07.396574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:07.396577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:07.396707 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:07.397060 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:07.397064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:07.397066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:07.397395 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:07.397665 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:07.397991 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 02:52:07.398260 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":260,"threads_started":1}
I20260929 02:52:07.399785 17575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:07.485158 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 17443
I20260929 02:52:07.496968 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 17586
Sep 29 02:52:07 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:52:07.513103 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:07.497940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.4.197.131:39509)
0929 02:52:07.498629 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:07.498636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:07.498663 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:52:07.498727 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:07.498733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:07.498992 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:07.499130 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:07.500455 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:07.501490 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:07.503028 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:07.503642 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:07.503740 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:07.507908 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:07.507919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:07.507933 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:07.507971 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:07.510645 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:07.511337 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:07.511343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:07.511346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:07.511473 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:07.511801 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:07.511805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:07.511807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:07.512151 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:07.512374 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:07.512582 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 02:52:07.512767 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":369,"threads_started":1}
I20260929 02:52:07.514297 17709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:07.596060 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 17586
I20260929 02:52:07.610705 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 17192
I20260929 02:52:07.611991 17254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:07.717499 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 17192
I20260929 02:52:07.734840 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33121
--webserver_interface=127.4.197.190
--webserver_port=34865
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:33121
--ranger_config_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:51961/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.197.148:51961/kms with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:08.051187 17726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:08.051489 17726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:08.051616 17726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:08.059973 17726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:52:08.060091 17726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:08.060176 17726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:08.060251 17726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:52:08.060313 17726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:52:08.070578 17726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.197.148:51961/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:33121
--ranger_config_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=34865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:08.071885 17726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:08.073220 17726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:08.084575 17731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:08.084662 17732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:08.084648 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:08.085382 17726 server_base.cc:1034] running on GCE node
I20260929 02:52:08.086104 17726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:08.087549 17726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:08.088761 17726 hybrid_clock.cc:648] HybridClock initialized: now 1790650328088714 us; error 33 us; skew 500 ppm
I20260929 02:52:08.089082 17726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:08 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:08.092008 17726 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:52:08.093348 17726 webserver.cc:460] Webserver started at http://127.4.197.190:34865/ using document root <none> and password file <none>
I20260929 02:52:08.093951 17726 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:08.094060 17726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:52:08 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:52:08.139744 17726 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:52:08.142814 17742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:08.144413 17726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:52:08.144591 17726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "eac360019a69471eac3e11b6463e709c"
format_stamp: "Formatted at 2026-09-29 02:52:04 on dist-test-slave-t6c1"
server_key: "28c962949dbc734220e9d22ba42a21d4"
server_key_iv: "cffb58c2f99b8891d154a68509253b8c"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:08.145540 17726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:08.173054 17726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:08.177224 17726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:08.177507 17726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:08.194432 17726 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:33121
I20260929 02:52:08.194451 17793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:33121 every 8 connection(s)
I20260929 02:52:08.195998 17726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:52:08.197271 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 17726
I20260929 02:52:08.198292 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:51961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:40725
--local_ip_for_outbound_sockets=127.4.197.129
--tserver_master_addrs=127.4.197.190:33121
--webserver_port=42601
--webserver_interface=127.4.197.129
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.197.148:51961/kms with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:52:08.205271 17794 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:52:08.209102 17794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: Bootstrap starting.
I20260929 02:52:08.229054 17794 log.cc:824] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:08.237833 17794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:52:08.238327 17794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: Bootstrap complete.
I20260929 02:52:08.247537 17794 raft_consensus.cc:348] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } }
I20260929 02:52:08.248456 17794 raft_consensus.cc:729] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac360019a69471eac3e11b6463e709c, State: Initialized, Role: FOLLOWER
I20260929 02:52:08.249135 17794 consensus_queue.cc:260] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } }
I20260929 02:52:08.249359 17794 raft_consensus.cc:388] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:52:08.249467 17794 raft_consensus.cc:482] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:52:08.249641 17794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:52:08.253816 17794 raft_consensus.cc:504] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } }
I20260929 02:52:08.254321 17794 leader_election.cc:302] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eac360019a69471eac3e11b6463e709c; no voters:
I20260929 02:52:08.255007 17794 leader_election.cc:288] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:52:08.255112 17798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:52:08.256270 17798 raft_consensus.cc:686] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [term 2 LEADER]: Becoming Leader. State: Replica: eac360019a69471eac3e11b6463e709c, State: Running, Role: LEADER
I20260929 02:52:08.256834 17798 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } }
I20260929 02:52:08.257324 17794 sys_catalog.cc:564] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:52:08.259456 17800 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [sys.catalog]: SysCatalogTable state changed. Reason: New leader eac360019a69471eac3e11b6463e709c. Latest consensus state: current_term: 2 leader_uuid: "eac360019a69471eac3e11b6463e709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } } }
I20260929 02:52:08.259832 17800 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:08.260365 17799 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eac360019a69471eac3e11b6463e709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac360019a69471eac3e11b6463e709c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33121 } } }
I20260929 02:52:08.260645 17799 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:08.270889 17804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:52:08.277755 17804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:52:08.279268 17804 catalog_manager.cc:1259] Loaded cluster ID: 75ada9262f444e5f84f690f5b5cf4fb9
I20260929 02:52:08.279448 17804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:52:08.283437 17804 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:52:08.285189 17804 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: Loaded TSK: 0
I20260929 02:52:08.287125 17804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:52:08.291082 17794 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 02:52:08.603055 17796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:08.603442 17796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:08.603634 17796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:08.614974 17796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:08.615175 17796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:08.615402 17796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:52:08.630432 17796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.197.148:51961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:40725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=42601
--tserver_master_addrs=127.4.197.190:33121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:08.632367 17796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:08.634200 17796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:08.657424 17832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:08.658864 17831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:08.659646 17834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:08.662050 17796 server_base.cc:1034] running on GCE node
I20260929 02:52:08.662971 17796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:08.673566 17796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:08.675033 17796 hybrid_clock.cc:648] HybridClock initialized: now 1790650328674947 us; error 82 us; skew 500 ppm
I20260929 02:52:08.675554 17796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:08 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:08.680289 17796 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:52:08.691365 17796 webserver.cc:460] Webserver started at http://127.4.197.129:42601/ using document root <none> and password file <none>
I20260929 02:52:08.692544 17796 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:08.692806 17796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:52:08 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:52:08.729163 17796 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:52:08.773603 17842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:08.781741 17796 fs_manager.cc:725] Time spent opening block manager: real 0.050s user 0.025s sys 0.016s
I20260929 02:52:08.783102 17796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "f72312b67598410a87ec34454e4ca4eb"
format_stamp: "Formatted at 2026-09-29 02:52:05 on dist-test-slave-t6c1"
server_key: "f72f5f598cacdb216c864020302c7fc9"
server_key_iv: "e29299bce0de7c85557fbe79ebadd566"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:08.789239 17796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:08.831676 17796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:08.837194 17796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:08.837589 17796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:08.842880 17796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:08.844992 17796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:08.845117 17796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:08.845290 17796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:08.845366 17796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:08.888751 17796 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:40725
I20260929 02:52:08.888803 17955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:40725 every 8 connection(s)
I20260929 02:52:08.890427 17796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:52:08.893204 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 17796
I20260929 02:52:08.894052 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:51961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:33609
--local_ip_for_outbound_sockets=127.4.197.130
--tserver_master_addrs=127.4.197.190:33121
--webserver_port=37645
--webserver_interface=127.4.197.130
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.197.148:51961/kms with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:52:08 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:08.933670 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:08.897091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33121 (local address 127.4.197.129:42529)
0929 02:52:08.903719 (+ 6628us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:08.903800 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:08.906621 (+ 2821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:08.911366 (+ 4745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:08.911381 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:08.912120 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:08.912847 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:08.912870 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:08.915270 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:08.915276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:08.917514 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:08.917542 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:08.919363 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:08.919371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:08.919690 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:08.920603 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:08.920639 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:08.923271 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:08.928271 (+ 5000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:08.928282 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:08.928303 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:08.928979 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:08.929571 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:08.929576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:08.929580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:08.929861 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:08.930975 (+ 1114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:08.930984 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:08.931416 (+ 432us) client_negotiation.cc:769] Sending connection context
0929 02:52:08.931716 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 02:52:08.932305 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6229,"thread_start_us":5978,"threads_started":1}
I20260929 02:52:08.936344 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:08.896984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:42529 (local address 127.4.197.190:33121)
0929 02:52:08.909353 (+ 12369us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:08.909375 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:08.909436 (+ 61us) server_negotiation.cc:407] Connection header received
0929 02:52:08.909987 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:08.910011 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:08.910584 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:08.911122 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:08.913097 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:08.915077 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:08.917709 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:08.918721 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:08.918954 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:08.923644 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:08.923657 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:08.923682 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:08.923745 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:08.928009 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:08.929174 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:08.929184 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:08.929205 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:08.929426 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:08.930005 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:08.930011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:08.930015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:08.930669 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:08.930868 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:08.934139 (+ 3271us) server_negotiation.cc:299] Negotiation successful
0929 02:52:08.934584 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12023,"thread_start_us":14447,"threads_started":1}
I20260929 02:52:08.940176 17956 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33121
I20260929 02:52:08.940481 17956 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:08.941191 17956 heartbeater.cc:511] Master 127.4.197.190:33121 requested a full tablet report, sending...
W20260929 02:52:08.944267 17956 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:33121 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 02:52:08.944487 17956 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:08.944790 17956 heartbeater.cc:511] Master 127.4.197.190:33121 requested a full tablet report, sending...
I20260929 02:52:08.946110 17956 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:08.946370 17956 heartbeater.cc:511] Master 127.4.197.190:33121 requested a full tablet report, sending...
W20260929 02:52:08.947492 17956 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:52:09.292418 17960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:09.292795 17960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:09.292994 17960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:09.300470 17960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:09.300565 17960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:09.300711 17960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:52:09.311074 17960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.197.148:51961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:33609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=37645
--tserver_master_addrs=127.4.197.190:33121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:09.312731 17960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:09.314390 17960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:09.329746 17975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:09.335338 17974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:09.335995 17977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:09.336563 17960 server_base.cc:1034] running on GCE node
I20260929 02:52:09.337292 17960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:09.338662 17960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:09.339943 17960 hybrid_clock.cc:648] HybridClock initialized: now 1790650329339827 us; error 114 us; skew 500 ppm
I20260929 02:52:09.340278 17960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:09 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650329, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:09.344607 17960 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:52:09.346617 17960 webserver.cc:460] Webserver started at http://127.4.197.130:37645/ using document root <none> and password file <none>
I20260929 02:52:09.347621 17960 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:09.347795 17960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:52:09 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650329, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:52:09.376446 17960 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:52:09.395584 17985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:09.397644 17960 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.006s sys 0.013s
I20260929 02:52:09.397908 17960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "9db8ca9195e9443e9e89fddc88f08c09"
format_stamp: "Formatted at 2026-09-29 02:52:06 on dist-test-slave-t6c1"
server_key: "e3807436f850217027c2112c701d843f"
server_key_iv: "898206ae630cb7694611105077043d19"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:09.399457 17960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:09.455770 17960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:09.461313 17960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:09.461625 17960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:09.462877 17960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:09.466912 17960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:09.467046 17960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 02:52:09.467154 17960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:09.467228 17960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:52:09 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650329, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:09.533566 17960 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:33609
I20260929 02:52:09.533617 18099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:33609 every 8 connection(s)
I20260929 02:52:09.535738 17960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:52:09.540107 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 17960
I20260929 02:52:09.540966 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.197.148:51961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:39509
--local_ip_for_outbound_sockets=127.4.197.131
--tserver_master_addrs=127.4.197.190:33121
--webserver_port=37687
--webserver_interface=127.4.197.131
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.197.148:51961/kms with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:52:09 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650329, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:09.588101 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:09.539781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33121 (local address 127.4.197.130:52477)
0929 02:52:09.562431 (+ 22650us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:09.562517 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:09.567111 (+ 4594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:09.567959 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:09.567969 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:09.568456 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:09.569128 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:09.569149 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:09.570789 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:09.570795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:09.573040 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:09.573057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:09.574351 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:09.574359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:09.574687 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:09.575600 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:09.575638 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:09.578585 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:09.582912 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:09.582948 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:09.582971 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:09.583654 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:09.584120 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:09.584125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:09.584129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:09.584405 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:09.585058 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:09.585067 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:09.585662 (+ 595us) client_negotiation.cc:769] Sending connection context
0929 02:52:09.585974 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 02:52:09.586609 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22217,"thread_start_us":22843,"threads_started":1}
I20260929 02:52:09.590049 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:09.539218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:52477 (local address 127.4.197.190:33121)
0929 02:52:09.562267 (+ 23049us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:09.562275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:09.562684 (+ 409us) server_negotiation.cc:407] Connection header received
0929 02:52:09.567452 (+ 4768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:09.567458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:09.567559 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:09.567659 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:09.569344 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:09.570559 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:09.573333 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:09.574040 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:09.574202 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:09.578962 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:09.578969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:09.578973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:09.579027 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:09.582669 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:09.583830 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:09.583835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:09.583840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:09.583997 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:09.584544 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:09.584549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:09.584553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:09.584915 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:09.589212 (+ 4297us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:09.589379 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 02:52:09.589690 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22856,"thread_start_us":25109,"threads_started":1}
I20260929 02:52:09.591634 18100 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33121
I20260929 02:52:09.591948 18100 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:09.592633 18100 heartbeater.cc:511] Master 127.4.197.190:33121 requested a full tablet report, sending...
I20260929 02:52:09.594545 17759 ts_manager.cc:194] Registered new tserver with Master: 9db8ca9195e9443e9e89fddc88f08c09 (127.4.197.130:33609)
I20260929 02:52:09.596756 17759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:52477
Sep 29 02:52:09 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650329, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:52:09.947642 17956 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:09.948088 17956 heartbeater.cc:511] Master 127.4.197.190:33121 requested a full tablet report, sending...
I20260929 02:52:09.950277 17759 ts_manager.cc:194] Registered new tserver with Master: f72312b67598410a87ec34454e4ca4eb (127.4.197.129:40725)
I20260929 02:52:09.951536 17759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:42529
W20260929 02:52:10.009922 18104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:10.010211 18104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:10.010365 18104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:10.017593 18104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:10.017722 18104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:10.017881 18104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:52:10.027957 18104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.197.148:51961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:39509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=37687
--tserver_master_addrs=127.4.197.190:33121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:10.029435 18104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:10.030786 18104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:10.047995 18110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:10.049165 18111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:10.067909 18113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:10.068543 18104 server_base.cc:1034] running on GCE node
I20260929 02:52:10.069346 18104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:10.070593 18104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:10.071822 18104 hybrid_clock.cc:648] HybridClock initialized: now 1790650330071743 us; error 62 us; skew 500 ppm
I20260929 02:52:10.072212 18104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:10 dist-test-slave-t6c1 krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650330, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:10.076393 18104 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:52:10.079474 18104 webserver.cc:460] Webserver started at http://127.4.197.131:37687/ using document root <none> and password file <none>
I20260929 02:52:10.080426 18104 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:10.080577 18104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:52:10 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650330, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:52:10.103623 18104 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:52:10.107018 18121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:10.108757 18104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:52:10.108969 18104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "e68a3d0c21984595a7013f011d9710e0"
format_stamp: "Formatted at 2026-09-29 02:52:07 on dist-test-slave-t6c1"
server_key: "e5d68be5add73b29729b974cdb7eec80"
server_key_iv: "49fac151d29f9294521cc42658125306"
server_key_version: "kuduclusterkey@0"
I20260929 02:52:10.110302 18104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:10.145805 18104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:10.150962 18104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:10.151319 18104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:10.153616 18104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:10.155417 18104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:10.155568 18104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 02:52:10.155712 18104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:10.155807 18104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:52:10 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650329, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for HTTP/127.4.197.148@KRBTEST.COM
I20260929 02:52:10.433106 18104 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:39509
I20260929 02:52:10.433142 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:39509 every 8 connection(s)
I20260929 02:52:10.434549 18104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:52:10.439594 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 18104
Sep 29 02:52:10 dist-test-slave-t6c1 krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650330, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:10.487069 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.440802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.4.197.190:33121)
0929 02:52:10.463454 (+ 22652us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:10.463460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:10.463484 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:52:10.463560 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:10.463566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:10.463682 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:10.463831 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:10.468133 (+ 4302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.469368 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.471527 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.472164 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.472357 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.476092 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:10.476098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:10.476102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:10.476146 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:10.479996 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:10.480922 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:10.480928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:10.480931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:10.481072 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:10.481544 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:10.481549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:10.481553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:10.481988 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:10.482255 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:10.483598 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0929 02:52:10.486640 (+ 3042us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22528,"thread_start_us":21917,"threads_started":1}
I20260929 02:52:10.491942 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.440380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:51573 (local address 127.4.197.190:33121)
0929 02:52:10.463041 (+ 22661us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:10.463048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:10.467639 (+ 4591us) server_negotiation.cc:407] Connection header received
0929 02:52:10.471696 (+ 4057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:10.471702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:10.471823 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:10.471953 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:10.473899 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.475115 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.477620 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.478332 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.478429 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.483063 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:10.483071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:10.483076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:10.483113 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:10.486422 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:10.488860 (+ 2438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:10.488866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:10.488870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:10.489028 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:10.489676 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:10.489681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:10.489685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:10.490070 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:10.490218 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:10.491289 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0929 02:52:10.491548 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22480,"thread_start_us":22815,"threads_started":1}
I20260929 02:52:10.494310 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.438283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33121 (local address 127.4.197.131:51573)
0929 02:52:10.466913 (+ 28630us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:10.466998 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:10.471332 (+ 4334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:10.472255 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:10.472270 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:10.472841 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:10.473641 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.473670 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.475327 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.475332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.477430 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.477462 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.478848 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.478856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.479182 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.479884 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:10.479916 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:10.482596 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:10.486806 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:10.486814 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:10.486834 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:10.487541 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:10.489212 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:10.489217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:10.489222 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:10.489532 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:10.490282 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:10.490292 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:10.490782 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 02:52:10.492150 (+ 1368us) client_negotiation.cc:240] Negotiation successful
0929 02:52:10.492773 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28270,"thread_start_us":28000,"threads_started":1}
I20260929 02:52:10.496260 18235 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33121
I20260929 02:52:10.496580 18235 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:10.497272 18235 heartbeater.cc:511] Master 127.4.197.190:33121 requested a full tablet report, sending...
I20260929 02:52:10.498629 17759 ts_manager.cc:194] Registered new tserver with Master: e68a3d0c21984595a7013f011d9710e0 (127.4.197.131:39509)
I20260929 02:52:10.499465 17759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:51573
I20260929 02:52:10.502527 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:52:10.520318 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.505568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.4.197.190:33121)
0929 02:52:10.505744 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:10.505750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:10.506276 (+ 526us) server_negotiation.cc:407] Connection header received
0929 02:52:10.506464 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:10.506470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:10.506546 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:10.506650 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:10.508881 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.509956 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.511575 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.512193 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.512327 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.514539 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:10.514545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:10.514550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:10.514587 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:10.517852 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:10.518440 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:10.518446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:10.518450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:10.518593 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:10.518893 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:10.518897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:10.518900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:10.519273 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:10.519498 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:10.519746 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 02:52:10.519971 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:52:10.524830 17759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:52:10.532361 17759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:52:10.585798 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.549379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.4.197.130:33609)
0929 02:52:10.574043 (+ 24664us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:10.574053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:10.575481 (+ 1428us) server_negotiation.cc:407] Connection header received
0929 02:52:10.575610 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:10.575621 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:10.575953 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:10.576147 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:10.577747 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.579570 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.582361 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.584049 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.584187 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.584431 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:10.585237 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 02:52:10.585418 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24440,"thread_start_us":24677,"threads_started":1}
I20260929 02:52:10.586575 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.548720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:33609 (local address 127.0.0.1:48614)
0929 02:52:10.574913 (+ 26193us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:10.574993 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:10.575199 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:10.577067 (+ 1868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:10.577076 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:10.577104 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:10.577518 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.577530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.579756 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.579762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.582151 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.582172 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.584664 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.584671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.584894 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.584956 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 02:52:10.586027 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0929 02:52:10.586249 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25860,"thread_start_us":28954,"threads_started":1}
I20260929 02:52:10.595683 18035 tablet_service.cc:1467] Processing CreateTablet for tablet 010138d79e4e4e2b9e0a9cca0de1f110 (DEFAULT_TABLE table=test-table [id=12d00beef0d4463785e1531073edc3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:10.597298 18035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010138d79e4e4e2b9e0a9cca0de1f110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:10.598958 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.584369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:39509 (local address 127.0.0.1:51368)
0929 02:52:10.586823 (+ 2454us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:10.586844 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:10.586987 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:10.592064 (+ 5077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:10.592068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:10.592094 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:10.592251 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.592262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.593968 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.593973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.596066 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.596090 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.598109 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.598116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.598320 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.598355 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:52:10.598448 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 02:52:10.598596 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2352,"mutex_wait_us":27,"thread_start_us":7081,"threads_started":1}
I20260929 02:52:10.599851 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.578452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.4.197.131:39509)
0929 02:52:10.591063 (+ 12611us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:10.591071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:10.591200 (+ 129us) server_negotiation.cc:407] Connection header received
0929 02:52:10.591310 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:10.591318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:10.591567 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:10.591771 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:10.592422 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.593849 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.596246 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.597665 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.597779 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.597997 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:10.599332 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0929 02:52:10.599503 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12388,"thread_start_us":12823,"threads_started":1}
I20260929 02:52:10.599918 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.548748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40725 (local address 127.0.0.1:49628)
0929 02:52:10.576786 (+ 28038us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:10.576805 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:10.576918 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:10.591336 (+ 14418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:10.591341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:10.591369 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:10.591750 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.591763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.593680 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.593685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.596014 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.596029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.599177 (+ 3148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.599183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.599349 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.599380 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:52:10.599454 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:52:10.599581 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27887,"mutex_wait_us":167,"thread_start_us":27734,"threads_started":1}
I20260929 02:52:10.600747 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.549499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.4.197.129:40725)
0929 02:52:10.590362 (+ 40863us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:10.590372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:10.590405 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:52:10.590512 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:10.590519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:10.590846 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:10.591101 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:10.591949 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.593536 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.596202 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.597562 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.597705 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.600198 (+ 2493us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:10.600273 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 02:52:10.600452 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40611,"thread_start_us":40457,"threads_started":1}
I20260929 02:52:10.603609 18170 tablet_service.cc:1467] Processing CreateTablet for tablet 010138d79e4e4e2b9e0a9cca0de1f110 (DEFAULT_TABLE table=test-table [id=12d00beef0d4463785e1531073edc3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:10.604192 17891 tablet_service.cc:1467] Processing CreateTablet for tablet 010138d79e4e4e2b9e0a9cca0de1f110 (DEFAULT_TABLE table=test-table [id=12d00beef0d4463785e1531073edc3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:10.604759 18100 heartbeater.cc:503] Master 127.4.197.190:33121 was elected leader, sending a full tablet report...
I20260929 02:52:10.605581 17891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010138d79e4e4e2b9e0a9cca0de1f110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:10.607019 18170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010138d79e4e4e2b9e0a9cca0de1f110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:10.618381 18263 tablet_bootstrap.cc:492] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: Bootstrap starting.
I20260929 02:52:10.621922 18263 tablet_bootstrap.cc:654] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:10.625365 18263 log.cc:824] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:10.626610 18264 tablet_bootstrap.cc:492] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb: Bootstrap starting.
I20260929 02:52:10.630435 18264 tablet_bootstrap.cc:654] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:10.631004 18265 tablet_bootstrap.cc:492] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: Bootstrap starting.
I20260929 02:52:10.631783 18264 log.cc:824] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:10.632058 18263 tablet_bootstrap.cc:492] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: No bootstrap required, opened a new log
I20260929 02:52:10.632373 18263 ts_tablet_manager.cc:1397] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260929 02:52:10.634444 18265 tablet_bootstrap.cc:654] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:10.635701 18265 log.cc:824] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:10.641166 18264 tablet_bootstrap.cc:492] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb: No bootstrap required, opened a new log
I20260929 02:52:10.641461 18264 ts_tablet_manager.cc:1397] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260929 02:52:10.642740 18265 tablet_bootstrap.cc:492] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: No bootstrap required, opened a new log
I20260929 02:52:10.643087 18265 ts_tablet_manager.cc:1397] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260929 02:52:10.642818 18263 raft_consensus.cc:348] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 } } peers { permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 } } peers { permanent_uuid: "e68a3d0c21984595a7013f011d9710e0" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39509 } }
I20260929 02:52:10.643374 18263 raft_consensus.cc:374] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:10.643468 18263 raft_consensus.cc:729] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db8ca9195e9443e9e89fddc88f08c09, State: Initialized, Role: FOLLOWER
I20260929 02:52:10.644111 18263 consensus_queue.cc:260] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 } } peers { permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 } } peers { permanent_uuid: "e68a3d0c21984595a7013f011d9710e0" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39509 } }
I20260929 02:52:10.652745 18264 raft_consensus.cc:348] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 } } peers { permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 } } peers { permanent_uuid: "e68a3d0c21984595a7013f011d9710e0" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39509 } }
I20260929 02:52:10.653368 18264 raft_consensus.cc:374] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:10.653481 18264 raft_consensus.cc:729] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72312b67598410a87ec34454e4ca4eb, State: Initialized, Role: FOLLOWER
I20260929 02:52:10.655053 18264 consensus_queue.cc:260] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 } } peers { permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 } } peers { permanent_uuid: "e68a3d0c21984595a7013f011d9710e0" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39509 } }
I20260929 02:52:10.655612 18265 raft_consensus.cc:348] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 } } peers { permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 } } peers { permanent_uuid: "e68a3d0c21984595a7013f011d9710e0" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39509 } }
I20260929 02:52:10.655925 18263 ts_tablet_manager.cc:1428] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: Time spent starting tablet: real 0.023s user 0.020s sys 0.004s
I20260929 02:52:10.656061 18265 raft_consensus.cc:374] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:10.656179 18265 raft_consensus.cc:729] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68a3d0c21984595a7013f011d9710e0, State: Initialized, Role: FOLLOWER
I20260929 02:52:10.656903 18265 consensus_queue.cc:260] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 } } peers { permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 } } peers { permanent_uuid: "e68a3d0c21984595a7013f011d9710e0" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39509 } }
I20260929 02:52:10.659452 17956 heartbeater.cc:503] Master 127.4.197.190:33121 was elected leader, sending a full tablet report...
I20260929 02:52:10.660506 18264 ts_tablet_manager.cc:1428] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260929 02:52:10.667308 18265 ts_tablet_manager.cc:1428] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: Time spent starting tablet: real 0.024s user 0.019s sys 0.000s
I20260929 02:52:10.667991 18235 heartbeater.cc:503] Master 127.4.197.190:33121 was elected leader, sending a full tablet report...
I20260929 02:52:10.701975 18271 raft_consensus.cc:482] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:10.702318 18271 raft_consensus.cc:504] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 } } peers { permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 } } peers { permanent_uuid: "e68a3d0c21984595a7013f011d9710e0" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39509 } }
I20260929 02:52:10.705533 18271 leader_election.cc:288] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f72312b67598410a87ec34454e4ca4eb (127.4.197.129:40725), 9db8ca9195e9443e9e89fddc88f08c09 (127.4.197.130:33609)
I20260929 02:52:10.711414 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.706259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:35211 (local address 127.4.197.130:33609)
0929 02:52:10.706422 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:10.706427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:10.706446 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:52:10.706507 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:10.706511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:10.706562 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:10.706636 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:10.707216 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.707906 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.709614 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.710526 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.710675 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.710746 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:10.711046 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 02:52:10.711177 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 02:52:10.711598 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.704803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:33609 (local address 127.4.197.131:35211)
0929 02:52:10.705570 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:10.705587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:10.705712 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:10.706686 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:10.706690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:10.706714 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:10.707042 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.707051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.708042 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.708047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.709471 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.709482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.710703 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.710708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.710865 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.710896 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:52:10.711022 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 02:52:10.711149 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":25,"thread_start_us":444,"threads_started":1}
I20260929 02:52:10.712239 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.705497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:55269 (local address 127.4.197.129:40725)
0929 02:52:10.705680 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:10.705686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:10.705705 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:52:10.705770 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:10.705775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:10.705850 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:10.705929 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:10.706799 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.707909 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.710146 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.711260 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.711398 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.711465 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:10.711893 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 02:52:10.712018 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:52:10.712551 18055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010138d79e4e4e2b9e0a9cca0de1f110" candidate_uuid: "e68a3d0c21984595a7013f011d9710e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db8ca9195e9443e9e89fddc88f08c09" is_pre_election: true
I20260929 02:52:10.712983 18055 raft_consensus.cc:2454] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e68a3d0c21984595a7013f011d9710e0 in term 0.
I20260929 02:52:10.713596 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.704598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40725 (local address 127.4.197.131:55269)
0929 02:52:10.705040 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:10.705059 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:10.705185 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:10.706433 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:10.706437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:10.706464 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:10.706635 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.706645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.708069 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.708090 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.709986 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:10.709999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.711563 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.711569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:10.711732 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.711762 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:52:10.711856 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:52:10.711981 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323}
I20260929 02:52:10.713572 18122 leader_election.cc:302] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9db8ca9195e9443e9e89fddc88f08c09, e68a3d0c21984595a7013f011d9710e0; no voters:
I20260929 02:52:10.714013 18271 raft_consensus.cc:2781] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:52:10.714200 18271 raft_consensus.cc:482] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:52:10.714313 18271 raft_consensus.cc:3037] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:10.714540 17911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010138d79e4e4e2b9e0a9cca0de1f110" candidate_uuid: "e68a3d0c21984595a7013f011d9710e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f72312b67598410a87ec34454e4ca4eb" is_pre_election: true
I20260929 02:52:10.715065 17911 raft_consensus.cc:2454] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e68a3d0c21984595a7013f011d9710e0 in term 0.
I20260929 02:52:10.716857 18271 raft_consensus.cc:504] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 } } peers { permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 } } peers { permanent_uuid: "e68a3d0c21984595a7013f011d9710e0" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39509 } }
I20260929 02:52:10.717391 18271 leader_election.cc:288] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [CANDIDATE]: Term 1 election: Requested vote from peers f72312b67598410a87ec34454e4ca4eb (127.4.197.129:40725), 9db8ca9195e9443e9e89fddc88f08c09 (127.4.197.130:33609)
I20260929 02:52:10.717808 17911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010138d79e4e4e2b9e0a9cca0de1f110" candidate_uuid: "e68a3d0c21984595a7013f011d9710e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f72312b67598410a87ec34454e4ca4eb"
I20260929 02:52:10.717871 18055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010138d79e4e4e2b9e0a9cca0de1f110" candidate_uuid: "e68a3d0c21984595a7013f011d9710e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db8ca9195e9443e9e89fddc88f08c09"
I20260929 02:52:10.717957 17911 raft_consensus.cc:3037] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:10.718070 18055 raft_consensus.cc:3037] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:10.720214 17911 raft_consensus.cc:2454] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e68a3d0c21984595a7013f011d9710e0 in term 1.
W20260929 02:52:10.720440 18236 tablet.cc:2369] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:52:10.720628 18125 leader_election.cc:302] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e68a3d0c21984595a7013f011d9710e0, f72312b67598410a87ec34454e4ca4eb; no voters:
I20260929 02:52:10.720904 18271 raft_consensus.cc:2781] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:52:10.721451 18271 raft_consensus.cc:686] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 1 LEADER]: Becoming Leader. State: Replica: e68a3d0c21984595a7013f011d9710e0, State: Running, Role: LEADER
I20260929 02:52:10.721601 18055 raft_consensus.cc:2454] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e68a3d0c21984595a7013f011d9710e0 in term 1.
I20260929 02:52:10.721935 18271 consensus_queue.cc:237] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 } } peers { permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 } } peers { permanent_uuid: "e68a3d0c21984595a7013f011d9710e0" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39509 } }
I20260929 02:52:10.726727 17759 catalog_manager.cc:5495] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e68a3d0c21984595a7013f011d9710e0 (127.4.197.131). New cstate: current_term: 1 leader_uuid: "e68a3d0c21984595a7013f011d9710e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e68a3d0c21984595a7013f011d9710e0" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39509 } health_report { overall_health: HEALTHY } } }
I20260929 02:52:10.752573 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.746650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.4.197.131:39509)
0929 02:52:10.746869 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:10.746875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:10.747000 (+ 125us) server_negotiation.cc:407] Connection header received
0929 02:52:10.747253 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:10.747258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:10.747337 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:10.747469 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:52:10.747864 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.748592 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.750117 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.750693 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.750808 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.751368 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:52:10.752021 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:52:10.752176 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:10.752219 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 02:52:10.752299 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:52:10.760717 17911 raft_consensus.cc:1261] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb [term 1 FOLLOWER]: Refusing update from remote peer e68a3d0c21984595a7013f011d9710e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:52:10.760720 18055 raft_consensus.cc:1261] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09 [term 1 FOLLOWER]: Refusing update from remote peer e68a3d0c21984595a7013f011d9710e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:52:10.761693 18273 consensus_queue.cc:1035] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f72312b67598410a87ec34454e4ca4eb" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:52:10.761999 18271 consensus_queue.cc:1035] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db8ca9195e9443e9e89fddc88f08c09" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:52:10.774580 18276 mvcc.cc:205] Tried to move back new op lower bound from 7334503754785882112 to 7334503754640732160. Current Snapshot: MvccSnapshot[applied={T|T < 7334503754785882112}]
I20260929 02:52:10.792377 18278 mvcc.cc:205] Tried to move back new op lower bound from 7334503754785882112 to 7334503754640732160. Current Snapshot: MvccSnapshot[applied={T|T < 7334503754785882112}]
I20260929 02:52:10.809381 17758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:52:10.809736 17758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:52:10.814226 17758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eac360019a69471eac3e11b6463e709c: Sending DeleteTablet for 3 replicas of tablet 010138d79e4e4e2b9e0a9cca0de1f110
I20260929 02:52:10.815497 17891 tablet_service.cc:1514] Processing DeleteTablet for tablet 010138d79e4e4e2b9e0a9cca0de1f110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:10 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:49628
I20260929 02:52:10.815690 18035 tablet_service.cc:1514] Processing DeleteTablet for tablet 010138d79e4e4e2b9e0a9cca0de1f110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:10 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:48614
I20260929 02:52:10.816542 18170 tablet_service.cc:1514] Processing DeleteTablet for tablet 010138d79e4e4e2b9e0a9cca0de1f110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:10 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:51368
I20260929 02:52:10.825577 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 17796
I20260929 02:52:10.848031 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:10.827391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.4.197.129:40725)
0929 02:52:10.827555 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:10.827562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:10.827642 (+ 80us) server_negotiation.cc:407] Connection header received
0929 02:52:10.827866 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:10.827872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:10.827954 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:10.828049 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:10.829893 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.831017 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.832907 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:10.833462 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:10.833591 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:10.839894 (+ 6303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:10.839915 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:10.839936 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:10.839990 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:10.843724 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:10.844629 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:10.844638 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:10.844641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:10.844854 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:10.845290 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:10.845294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:10.845297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:10.845704 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:10.846338 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:10.847460 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0929 02:52:10.847679 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:52:10.849045 17951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:10.855480 18288 tablet_replica.cc:330] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: stopping tablet replica
I20260929 02:52:10.856166 18288 raft_consensus.cc:2229] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:52:10.856851 18287 tablet_replica.cc:330] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: stopping tablet replica
I20260929 02:52:10.857674 18288 raft_consensus.cc:2258] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:10.858310 18287 raft_consensus.cc:2229] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:52:10.859258 18287 raft_consensus.cc:2258] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:10.861810 18287 ts_tablet_manager.cc:1905] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:52:10.872933 18288 ts_tablet_manager.cc:1905] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:52:10.877455 18287 ts_tablet_manager.cc:1918] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:52:10.877800 18287 log.cc:1196] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/010138d79e4e4e2b9e0a9cca0de1f110
I20260929 02:52:10.878418 18287 ts_tablet_manager.cc:1939] T 010138d79e4e4e2b9e0a9cca0de1f110 P 9db8ca9195e9443e9e89fddc88f08c09: Deleting consensus metadata
I20260929 02:52:10.880548 17743 catalog_manager.cc:4841] TS 9db8ca9195e9443e9e89fddc88f08c09 (127.4.197.130:33609): tablet 010138d79e4e4e2b9e0a9cca0de1f110 (table test-table [id=12d00beef0d4463785e1531073edc3e4]) successfully deleted
I20260929 02:52:10.882478 18288 ts_tablet_manager.cc:1918] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:52:10.882674 18288 log.cc:1196] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/010138d79e4e4e2b9e0a9cca0de1f110
I20260929 02:52:10.883244 18288 ts_tablet_manager.cc:1939] T 010138d79e4e4e2b9e0a9cca0de1f110 P e68a3d0c21984595a7013f011d9710e0: Deleting consensus metadata
I20260929 02:52:10.884995 17743 catalog_manager.cc:4841] TS e68a3d0c21984595a7013f011d9710e0 (127.4.197.131:39509): tablet 010138d79e4e4e2b9e0a9cca0de1f110 (table test-table [id=12d00beef0d4463785e1531073edc3e4]) successfully deleted
I20260929 02:52:10.992841 18286 tablet_replica.cc:330] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb: stopping tablet replica
I20260929 02:52:10.993289 18286 raft_consensus.cc:2229] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:52:10.993774 18286 raft_consensus.cc:2258] T 010138d79e4e4e2b9e0a9cca0de1f110 P f72312b67598410a87ec34454e4ca4eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:10.994735 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 17796
W20260929 02:52:11.007678 17746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.197.129:40725 (error 108)
I20260929 02:52:11.008373 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 17960
I20260929 02:52:11.009555 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:11.008961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40725 (local address 127.0.0.1:49654)
0929 02:52:11.009241 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:11.009361 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260929 02:52:11.009959 17746 catalog_manager.cc:4584] TS f72312b67598410a87ec34454e4ca4eb (127.4.197.129:40725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 010138d79e4e4e2b9e0a9cca0de1f110: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40725: connect: Connection refused (error 111)
I20260929 02:52:11.027383 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:11.009145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.4.197.130:33609)
0929 02:52:11.009328 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:11.009334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:11.009358 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:52:11.009531 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:11.009538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:11.009634 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:11.009747 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:11.011392 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:11.012179 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:11.013903 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:11.014325 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:11.014453 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:11.021263 (+ 6810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:11.021278 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:11.021293 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:11.021336 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:11.024326 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:11.025087 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:11.025095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:11.025098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:11.025265 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:11.025666 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:11.025671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:11.025673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:11.026070 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:11.026376 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:11.026708 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 02:52:11.026980 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:52:11.028162 18095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:11.063925 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:11.063345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40725 (local address 127.0.0.1:49670)
0929 02:52:11.063586 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:11.063723 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 02:52:11.065192 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:11.064744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40725 (local address 127.0.0.1:49686)
0929 02:52:11.064914 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:11.065032 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:52:11.119493 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 17960
I20260929 02:52:11.125856 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:11.125278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40725 (local address 127.0.0.1:49700)
0929 02:52:11.125531 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:11.125645 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 02:52:11.127158 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:11.126683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40725 (local address 127.0.0.1:49712)
0929 02:52:11.126862 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:11.126991 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 02:52:11.133248 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 18104
I20260929 02:52:11.146741 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:11.134201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.4.197.131:39509)
0929 02:52:11.134391 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:11.134398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:11.134415 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:52:11.134470 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:11.134473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:11.134529 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:11.134607 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:11.135905 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:11.136707 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:11.137984 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:11.138430 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:11.138540 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:11.140932 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:11.140946 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:11.140961 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:11.140999 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:11.144006 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:11.144714 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:11.144721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:11.144724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:11.144885 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:11.145228 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:11.145232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:11.145234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:11.145577 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:11.145861 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:11.146169 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 02:52:11.146381 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:52:11.147560 18230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:11.207999 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:11.207460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40725 (local address 127.0.0.1:49714)
0929 02:52:11.207679 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:11.207794 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 02:52:11.209128 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:11.208725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40725 (local address 127.0.0.1:49720)
0929 02:52:11.208940 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:11.209015 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 02:52:11.268730 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 18104
I20260929 02:52:11.284143 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 17726
I20260929 02:52:11.285353 17789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:11.396005 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:11.395465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40725 (local address 127.0.0.1:49724)
0929 02:52:11.395673 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:11.395798 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 02:52:11.396057 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 17726
I20260929 02:52:11.415737 4886 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 02:52:11.751240 4886 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 02:52:11.751434 4886 mini_ranger.cc:66] Stopping Ranger...
I20260929 02:52:12.086941 4886 mini_ranger.cc:68] Stopped Ranger
2026-09-29 02:52:12.087 UTC [16778] LOG: received smart shutdown request
2026-09-29 02:52:12.089 UTC [16778] LOG: background worker "logical replication launcher" (PID 16787) exited with exit code 1
2026-09-29 02:52:12.090 UTC [16782] LOG: shutting down
2026-09-29 02:52:12.242 UTC [16778] LOG: database system is shut down
2026-09-29T02:52:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50443 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:52:12 dist-test-slave-t6c1 krb5kdc[18302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:52:12 dist-test-slave-t6c1 krb5kdc[18302](info): set up 2 sockets
Sep 29 02:52:12 dist-test-slave-t6c1 krb5kdc[18302](info): commencing operation
krb5kdc: starting...
W20260929 02:52:13.359220 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:52:13 dist-test-slave-t6c1 krb5kdc[18302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:52:13.574507 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33459
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:33459,127.4.197.189:38123,127.4.197.188:35793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:13.899163 18315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:13.899438 18315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:13.899570 18315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:13.906687 18315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:52:13.906778 18315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:13.906849 18315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:13.906941 18315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:52:13.906997 18315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:52:13.916487 18315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:33459,127.4.197.189:38123,127.4.197.188:35793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:13.917687 18315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:13.918962 18315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:13.930480 18323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:13.930464 18321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:13.931087 18320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:13.931391 18315 server_base.cc:1034] running on GCE node
I20260929 02:52:13.932175 18315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:13.933624 18315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:13.934823 18315 hybrid_clock.cc:648] HybridClock initialized: now 1790650333934763 us; error 49 us; skew 500 ppm
I20260929 02:52:13.935182 18315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:13 dist-test-slave-t6c1 krb5kdc[18302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:13.938163 18315 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:52:13.939682 18315 webserver.cc:460] Webserver started at http://127.4.197.190:33055/ using document root <none> and password file <none>
I20260929 02:52:13.940371 18315 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:13.940477 18315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:13.940769 18315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:13.943817 18315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "8881c191fccc433989aee5664b380dff"
format_stamp: "Formatted at 2026-09-29 02:52:13 on dist-test-slave-t6c1"
server_key: "113a1b834782a1a2191368fec989050d"
server_key_iv: "75d329a26766b81a108bca3434f075d5"
server_key_version: "encryptionkey@0"
I20260929 02:52:13.944646 18315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "8881c191fccc433989aee5664b380dff"
format_stamp: "Formatted at 2026-09-29 02:52:13 on dist-test-slave-t6c1"
server_key: "113a1b834782a1a2191368fec989050d"
server_key_iv: "75d329a26766b81a108bca3434f075d5"
server_key_version: "encryptionkey@0"
I20260929 02:52:13.949805 18315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:52:13.953385 18330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:13.954855 18315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:52:13.955061 18315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "8881c191fccc433989aee5664b380dff"
format_stamp: "Formatted at 2026-09-29 02:52:13 on dist-test-slave-t6c1"
server_key: "113a1b834782a1a2191368fec989050d"
server_key_iv: "75d329a26766b81a108bca3434f075d5"
server_key_version: "encryptionkey@0"
I20260929 02:52:13.955278 18315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:13.970772 18315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:13.975118 18315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:13.975389 18315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:13.992909 18315 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:33459
I20260929 02:52:13.992933 18381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:33459 every 8 connection(s)
I20260929 02:52:13.994273 18315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:52:13.995337 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 18315
I20260929 02:52:13.995563 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:52:13.996374 4886 external_mini_cluster.cc:1428] Setting key 3b1031a96da88b88333942d4e3a32f27
I20260929 02:52:13.999104 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:14.007079 18382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.197.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.189@KRBTEST.COM" created.
Sep 29 02:52:14 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:14.029086 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.011617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.4.197.190:33459)
0929 02:52:14.012267 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:14.012275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:14.012323 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:52:14.013481 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:14.013488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:14.013779 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:14.013994 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:14.015076 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.016942 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.018373 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.018905 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.019093 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.022558 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:14.022565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:14.022576 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:14.022604 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:14.025139 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.025742 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.025749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.025751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.025894 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.026235 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.026239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.026242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.026702 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:14.026879 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:14.027383 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 02:52:14.027697 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":466,"thread_start_us":295,"threads_started":1}
I20260929 02:52:14.029103 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.010322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33459 (local address 127.0.0.1:53548)
0929 02:52:14.011095 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.011140 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:14.013072 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:14.013994 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:14.013999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:14.014531 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:14.014868 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.014885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.017112 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.017116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.018235 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.018246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.019184 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.019193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.019376 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.020040 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:14.020068 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:14.022361 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:14.025256 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:14.025267 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:14.025278 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:14.025631 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:14.025990 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:14.025995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:14.025999 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:14.026136 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:14.026826 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:14.026833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:14.027048 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 02:52:14.027269 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 02:52:14.027818 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":152,"thread_start_us":315,"threads_started":1}
I20260929 02:52:14.031960 18382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:52:14.033123 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.032675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50000)
0929 02:52:14.032859 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.032979 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 02:52:14.033438 18332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
I20260929 02:52:14.035120 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.034699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50016)
0929 02:52:14.034862 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.034968 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:52:14.035379 18382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:38123: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab.
Entry for principal kudu/127.4.197.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab.
I20260929 02:52:14.045499 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.189:38123
--webserver_interface=127.4.197.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab
--principal=kudu/127.4.197.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:33459,127.4.197.189:38123,127.4.197.188:35793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:52:14.084741 18382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } attempt: 1
I20260929 02:52:14.086155 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.085551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50020)
0929 02:52:14.085812 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.085967 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 02:52:14.087345 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.086890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50022)
0929 02:52:14.087070 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.087161 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 02:52:14.087543 18382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:38123: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
I20260929 02:52:14.155829 18382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } attempt: 2
I20260929 02:52:14.157159 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.156599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50024)
0929 02:52:14.156867 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.156993 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:52:14.158221 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.157805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50026)
0929 02:52:14.157963 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.158060 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 02:52:14.158443 18382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:38123: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
I20260929 02:52:14.249737 18382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } attempt: 3
I20260929 02:52:14.251118 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.250499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50040)
0929 02:52:14.250754 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.250896 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 02:52:14.252287 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.251881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50044)
0929 02:52:14.252041 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.252120 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 02:52:14.252496 18382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:38123: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
W20260929 02:52:14.368149 18389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:14.368445 18389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:14.368543 18389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:14.376077 18389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:52:14.376202 18389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:14.376255 18389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:14.376313 18389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:52:14.376355 18389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:52:14.386464 18389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:33459,127.4.197.189:38123,127.4.197.188:35793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab
--principal=kudu/127.4.197.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.189:38123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.197.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:14.387876 18389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:14.389147 18389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 02:52:14.395699 18382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } attempt: 4
I20260929 02:52:14.396761 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.396253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50048)
0929 02:52:14.396544 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.396629 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260929 02:52:14.397733 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.397402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50064)
0929 02:52:14.397538 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.397614 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:52:14.397936 18382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:38123: Network error: Client connection negotiation failed: client connection to 127.4.197.189:38123: connect: Connection refused (error 111)
W20260929 02:52:14.400235 18394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:14.400328 18395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:14.400358 18397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:14.401163 18389 server_base.cc:1034] running on GCE node
I20260929 02:52:14.401875 18389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:14.403203 18389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:14.404384 18389 hybrid_clock.cc:648] HybridClock initialized: now 1790650334404338 us; error 43 us; skew 500 ppm
I20260929 02:52:14.404673 18389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:14 dist-test-slave-t6c1 krb5kdc[18302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:14.407455 18389 init.cc:377] Logged in from keytab as kudu/127.4.197.189@KRBTEST.COM (short username kudu)
I20260929 02:52:14.408772 18389 webserver.cc:460] Webserver started at http://127.4.197.189:41039/ using document root <none> and password file <none>
I20260929 02:52:14.409380 18389 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:14.409479 18389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:14.409754 18389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:14.412487 18389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/data/instance:
uuid: "b63494e451024f079447f87b6d1ff8e6"
format_stamp: "Formatted at 2026-09-29 02:52:14 on dist-test-slave-t6c1"
server_key: "2663d35361ce61fed7be603734c77e94"
server_key_iv: "b79568989b5663c5ca4f811660b29eb1"
server_key_version: "encryptionkey@0"
I20260929 02:52:14.413199 18389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/wal/instance:
uuid: "b63494e451024f079447f87b6d1ff8e6"
format_stamp: "Formatted at 2026-09-29 02:52:14 on dist-test-slave-t6c1"
server_key: "2663d35361ce61fed7be603734c77e94"
server_key_iv: "b79568989b5663c5ca4f811660b29eb1"
server_key_version: "encryptionkey@0"
I20260929 02:52:14.417739 18389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:52:14.420846 18404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:14.422012 18389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:52:14.422161 18389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/wal
uuid: "b63494e451024f079447f87b6d1ff8e6"
format_stamp: "Formatted at 2026-09-29 02:52:14 on dist-test-slave-t6c1"
server_key: "2663d35361ce61fed7be603734c77e94"
server_key_iv: "b79568989b5663c5ca4f811660b29eb1"
server_key_version: "encryptionkey@0"
I20260929 02:52:14.422327 18389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:14.446672 18389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:14.450763 18389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:14.451077 18389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:14.468770 18389 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.189:38123
I20260929 02:52:14.468794 18455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.189:38123 every 8 connection(s)
I20260929 02:52:14.470124 18389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
I20260929 02:52:14.473922 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:14.476035 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 18389
I20260929 02:52:14.476255 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-1/wal/instance
I20260929 02:52:14.476799 4886 external_mini_cluster.cc:1428] Setting key 0c49f9794be44bd4fd944a1d1eed54be
I20260929 02:52:14.482849 18456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.197.188@KRBTEST.COM; defaulting to no policy
Sep 29 02:52:14 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.188@KRBTEST.COM" created.
I20260929 02:52:14.503242 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.486113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.4.197.190:33459)
0929 02:52:14.486304 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:14.486310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:14.487917 (+ 1607us) server_negotiation.cc:407] Connection header received
0929 02:52:14.489828 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:14.489835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:14.489916 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:14.490004 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:14.491644 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.492323 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.493885 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.494307 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.494419 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.497672 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:14.497677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:14.497680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:14.497717 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:14.500261 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.500956 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.500960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.500963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.501080 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.501489 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.501492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.501495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.501777 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:14.501929 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:14.502634 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 02:52:14.502889 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:52:14.504165 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.486857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33459 (local address 127.0.0.1:53564)
0929 02:52:14.487729 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.487770 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:14.489527 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:14.490271 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:14.490281 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:14.490822 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:14.491467 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.491486 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.492481 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.492485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.493727 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.493741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.494535 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.494544 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.494751 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.495407 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:14.495430 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:14.497452 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:14.500399 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:14.500408 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:14.500425 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:14.500832 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:14.501181 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:14.501187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:14.501191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:14.501374 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:14.501910 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:14.501918 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:14.502256 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 02:52:14.502503 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 02:52:14.503045 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":329,"threads_started":1}
I20260929 02:52:14.505854 18456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:52:14 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
I20260929 02:52:14.519888 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.506597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50066)
0929 02:52:14.506748 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.506762 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:14.506842 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:14.507853 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:14.507862 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:14.508289 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:14.508458 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.508469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.509749 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.509754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.510945 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.510958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.511728 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.511734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.511915 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.512451 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:14.512471 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:14.514776 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:14.517689 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:14.517693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:14.517695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:14.518065 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:14.518379 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:14.518382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:14.518385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:14.518520 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:14.519139 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:14.519144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:14.519290 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 02:52:14.519424 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:52:14.519607 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 02:52:14.520054 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.506853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.4.197.189:38123)
0929 02:52:14.507459 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:14.507465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:14.507490 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:52:14.507529 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:14.507534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:14.507701 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:14.507816 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:14.508609 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.509614 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.511060 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.511570 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.511661 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.515006 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:14.515017 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:14.515030 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:14.515063 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:14.517574 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.518170 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.518176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.518178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.518285 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.518620 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.518623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.518625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.519002 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:14.519172 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:14.519535 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 02:52:14.519707 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":347,"threads_started":1}
I20260929 02:52:14.522205 18456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab.
Entry for principal kudu/127.4.197.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab.
I20260929 02:52:14.523375 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.522891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37430)
0929 02:52:14.523074 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.523199 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 02:52:14.523659 18405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
I20260929 02:52:14.524590 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.188:35793
--webserver_interface=127.4.197.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab
--principal=kudu/127.4.197.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:33459,127.4.197.189:38123,127.4.197.188:35793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:52:14.525281 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.524888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37434)
0929 02:52:14.525036 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.525136 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:52:14.525527 18456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:35793: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
I20260929 02:52:14.574909 18456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } attempt: 1
I20260929 02:52:14.576292 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.575753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37442)
0929 02:52:14.575955 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.576115 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 02:52:14.577374 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.576981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37448)
0929 02:52:14.577162 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.577244 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:52:14.577610 18456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:35793: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
I20260929 02:52:14.645866 18456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } attempt: 2
I20260929 02:52:14.647112 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.646602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37458)
0929 02:52:14.646813 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.646906 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 02:52:14.648029 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.647696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37472)
0929 02:52:14.647846 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.647915 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 02:52:14.648227 18456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:35793: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
I20260929 02:52:14.697194 18382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } attempt: 5
Sep 29 02:52:14 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
I20260929 02:52:14.711779 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.697991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.4.197.189:38123)
0929 02:52:14.698195 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:14.698200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:14.698220 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:52:14.698390 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:14.698394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:14.698474 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:14.698564 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:14.699568 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.700258 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.701961 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.702435 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.702532 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.706366 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:14.706372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:14.706375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:14.706424 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:14.709020 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.709777 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.709782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.709784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.709913 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.710284 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.710288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.710290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.710619 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:14.710759 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:14.711230 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 02:52:14.711461 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:52:14.711797 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.697857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50080)
0929 02:52:14.698095 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.698117 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:14.698248 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:14.698621 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:14.698626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:14.699158 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:14.699410 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.699424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.700395 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.700401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.701786 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.701803 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.702589 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.702594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.702790 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.703493 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:14.703520 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:14.706138 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:14.709202 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:14.709209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:14.709213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:14.709649 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:14.710029 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:14.710035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:14.710039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:14.710179 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:14.710738 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:14.710743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:14.710902 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 02:52:14.711113 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 02:52:14.711450 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 02:52:14.712788 18382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:52:14.713833 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.713447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37488)
0929 02:52:14.713621 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.713704 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 02:52:14.714701 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.714394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37496)
0929 02:52:14.714522 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.714609 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 02:52:14.714870 18382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:35793: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
I20260929 02:52:14.739449 18456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } attempt: 3
I20260929 02:52:14.740654 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.740161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37510)
0929 02:52:14.740383 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.740480 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 02:52:14.741912 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.741433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37526)
0929 02:52:14.741700 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.741788 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260929 02:52:14.742137 18456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:35793: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
I20260929 02:52:14.766146 18382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } attempt: 1
I20260929 02:52:14.767441 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.766877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37542)
0929 02:52:14.767124 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.767257 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 02:52:14.768658 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.768234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37550)
0929 02:52:14.768397 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.768502 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 02:52:14.768937 18382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:35793: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
I20260929 02:52:14.837260 18382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } attempt: 2
I20260929 02:52:14.838579 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.837985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37566)
0929 02:52:14.838247 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.838378 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 02:52:14.839679 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.839274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37576)
0929 02:52:14.839432 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.839525 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:52:14.839891 18382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:35793: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
W20260929 02:52:14.852180 18463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:14.852483 18463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:14.852607 18463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:14.859881 18463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:52:14.859984 18463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:14.860057 18463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:14.860134 18463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:52:14.860193 18463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:52:14.869968 18463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:33459,127.4.197.189:38123,127.4.197.188:35793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab
--principal=kudu/127.4.197.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.188:35793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.197.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:14.871312 18463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:14.872691 18463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:14.885126 18468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:14.885207 18469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:14.885341 18456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } attempt: 4
W20260929 02:52:14.885254 18471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:14.885797 18463 server_base.cc:1034] running on GCE node
I20260929 02:52:14.886519 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.886011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37582)
0929 02:52:14.886262 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.886353 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 02:52:14.886879 18463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:14.887821 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.887371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37592)
0929 02:52:14.887557 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.887663 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 02:52:14.888090 18456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:35793: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
I20260929 02:52:14.888811 18463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:14.890062 18463 hybrid_clock.cc:648] HybridClock initialized: now 1790650334890002 us; error 47 us; skew 500 ppm
I20260929 02:52:14.890479 18463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:14 dist-test-slave-t6c1 krb5kdc[18302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:14.893535 18463 init.cc:377] Logged in from keytab as kudu/127.4.197.188@KRBTEST.COM (short username kudu)
I20260929 02:52:14.895107 18463 webserver.cc:460] Webserver started at http://127.4.197.188:41913/ using document root <none> and password file <none>
I20260929 02:52:14.895817 18463 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:14.895939 18463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:14.896225 18463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:14.899253 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/data/instance:
uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14"
format_stamp: "Formatted at 2026-09-29 02:52:14 on dist-test-slave-t6c1"
server_key: "db21db60f4ea1589ef00ffb2ac208454"
server_key_iv: "9cbdc86d24a44bd4f9bda2f2092fae3d"
server_key_version: "encryptionkey@0"
I20260929 02:52:14.900045 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/wal/instance:
uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14"
format_stamp: "Formatted at 2026-09-29 02:52:14 on dist-test-slave-t6c1"
server_key: "db21db60f4ea1589ef00ffb2ac208454"
server_key_iv: "9cbdc86d24a44bd4f9bda2f2092fae3d"
server_key_version: "encryptionkey@0"
I20260929 02:52:14.905198 18463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:52:14.908560 18478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:14.909866 18463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:52:14.910023 18463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/wal
uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14"
format_stamp: "Formatted at 2026-09-29 02:52:14 on dist-test-slave-t6c1"
server_key: "db21db60f4ea1589ef00ffb2ac208454"
server_key_iv: "9cbdc86d24a44bd4f9bda2f2092fae3d"
server_key_version: "encryptionkey@0"
I20260929 02:52:14.910198 18463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:14.931882 18463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:14.936110 18463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:14.936369 18463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:14.946090 18382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } attempt: 3
I20260929 02:52:14.947296 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.946784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37604)
0929 02:52:14.946989 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.947100 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 02:52:14.948351 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.947967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37618)
0929 02:52:14.948150 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.948228 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 02:52:14.948621 18382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:35793: Network error: Client connection negotiation failed: client connection to 127.4.197.188:35793: connect: Connection refused (error 111)
I20260929 02:52:14.953950 18463 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.188:35793
I20260929 02:52:14.953979 18529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.188:35793 every 8 connection(s)
I20260929 02:52:14.955332 18463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
I20260929 02:52:14.959090 18530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:14.965293 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 18463
I20260929 02:52:14.965499 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650137986910-4886-0/minicluster-data/master-2/wal/instance
I20260929 02:52:14.966071 4886 external_mini_cluster.cc:1428] Setting key f10bf14adec03fa3c52ad598860aae7e
I20260929 02:52:14.965937 18530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:52:14 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
Sep 29 02:52:14 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
I20260929 02:52:14.985651 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.969035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.4.197.190:33459)
0929 02:52:14.969214 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:14.969220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:14.969947 (+ 727us) server_negotiation.cc:407] Connection header received
0929 02:52:14.971364 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:14.971370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:14.971448 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:14.971531 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:14.972849 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.973623 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.975150 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.975588 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.975702 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.979598 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:14.979606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:14.979611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:14.979656 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:14.982275 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.983233 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.983237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.983239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.983379 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.983850 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.983852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.983854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.984099 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:14.984218 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:14.985060 (+ 842us) server_negotiation.cc:299] Negotiation successful
0929 02:52:14.985286 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:52:14.986690 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.969086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33459 (local address 127.0.0.1:53568)
0929 02:52:14.969783 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.969828 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:14.971107 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:14.971607 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:14.971615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:14.972272 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:14.972657 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.972674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.973777 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.973780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.975013 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.975024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.975816 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.975824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.976086 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.976818 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:14.976845 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:14.979382 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:14.982414 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:14.982426 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:14.982441 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:14.983128 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:14.983482 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:14.983486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:14.983490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:14.983755 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:14.984217 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:14.984225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:14.984689 (+ 464us) client_negotiation.cc:769] Sending connection context
0929 02:52:14.984952 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 02:52:14.985416 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":20,"thread_start_us":339,"threads_started":1}
I20260929 02:52:14.987133 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.968121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.4.197.188:35793)
0929 02:52:14.968970 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:14.968987 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:14.969036 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:52:14.971103 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:14.971126 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:14.971372 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:14.971565 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:14.973094 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.974326 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.975703 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.976316 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.976426 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.980676 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:14.980685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:14.980699 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:14.980738 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:14.983908 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.984685 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.984707 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.984709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.984837 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:14.985233 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:14.985238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:14.985241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:14.985674 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:14.985970 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:14.986476 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 02:52:14.986729 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":618,"thread_start_us":375,"threads_started":1}
I20260929 02:52:14.988308 18530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:52:14 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
Sep 29 02:52:14 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:15.003177 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.988941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:38123 (local address 127.0.0.1:50086)
0929 02:52:14.989102 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:14.989120 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:14.989228 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:14.989531 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:14.989534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:14.989954 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:14.990151 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.990163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.991575 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.991579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.992867 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:14.992882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.993820 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.993826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:14.994064 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.994733 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:14.994756 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:14.997337 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:15.000484 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:15.000489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:15.000494 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:15.001044 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:15.001536 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:15.001542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:15.001546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:15.001739 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:15.002362 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:15.002367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:15.002515 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 02:52:15.002633 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 02:52:15.002886 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 02:52:15.003393 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.989035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.4.197.189:38123)
0929 02:52:14.989216 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:14.989221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:14.989240 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:52:14.989329 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:14.989334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:14.989412 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:14.989515 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:14.990354 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.991423 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.993027 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.993651 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.993781 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:14.997538 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:14.997543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:14.997547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:14.997581 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:15.000303 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.001218 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.001224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.001228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.001408 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.001863 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.001868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.001871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.002238 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:15.002374 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:15.002750 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 02:52:15.003026 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:52:15.004215 18530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:52:15.006808 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:14.990064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.4.197.190:33459)
0929 02:52:14.990241 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:14.990247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:14.990265 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:52:14.990344 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:14.990350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:14.990410 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:14.990477 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:14.991967 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.993095 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.994664 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:14.995320 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:14.995448 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.000478 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:15.000485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:15.000490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:15.000532 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:15.004059 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.004810 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.004816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.004819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.004981 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.005411 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.005417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.005420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.005704 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:15.005951 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:15.006310 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 02:52:15.006529 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 02:52:15 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
Sep 29 02:52:15 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
I20260929 02:52:15.017411 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:15.004965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37636)
0929 02:52:15.005091 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:15.005108 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:15.005214 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:15.005459 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:15.005463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:15.005826 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:15.005949 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:15.005957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.006880 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.006885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:15.007965 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:15.007976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.008743 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.008749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:15.009008 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.009559 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:15.009575 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:15.011730 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:15.015045 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:15.015050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:15.015054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:15.015509 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:15.015885 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:15.015890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:15.015894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:15.016045 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:15.016621 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:15.016625 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:15.016760 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 02:52:15.016867 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 02:52:15.017130 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":2}
I20260929 02:52:15.017716 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:15.004906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.4.197.188:35793)
0929 02:52:15.005064 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:15.005070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:15.005163 (+ 93us) server_negotiation.cc:407] Connection header received
0929 02:52:15.005303 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:15.005308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:15.005362 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:15.005433 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:15.006067 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.006750 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.008084 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.008609 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.008742 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.011937 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:15.011942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:15.011946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:15.011977 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:15.014901 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.015649 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.015654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.015656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.015783 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.016154 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.016157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.016160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.016504 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:15.016662 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:15.017004 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 02:52:15.017316 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":48}
I20260929 02:52:15.022737 18530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14: Bootstrap starting.
I20260929 02:52:15.023375 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:15.008458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.4.197.189:38123)
0929 02:52:15.008591 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:15.008595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:15.008609 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:52:15.008763 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:15.008767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:15.008837 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:15.008934 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:15.010175 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.011072 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.012547 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.013251 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.013390 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.018104 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:15.018109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:15.018113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:15.018143 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:15.020812 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.021549 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.021552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.021563 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.021705 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.022106 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.022110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.022112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.022363 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:15.022586 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:15.022896 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 02:52:15.023111 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:52:15.026346 18530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:15.027604 18530 log.cc:824] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:15.028086 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:52:15.030349 18530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14: No bootstrap required, opened a new log
I20260929 02:52:15.041256 18530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.041764 18530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:15.041904 18530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8d5787066c4c3cb9eb8ec0b0af3f14, State: Initialized, Role: FOLLOWER
I20260929 02:52:15.043272 18530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.049218 18543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } } }
I20260929 02:52:15.049834 18543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:52:15.049700 18530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:52:15.054098 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:15.033337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.4.197.189:38123)
0929 02:52:15.035120 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:15.035126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:15.035605 (+ 479us) server_negotiation.cc:407] Connection header received
0929 02:52:15.035830 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:15.035835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:15.035907 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:15.035968 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:15.038342 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.039464 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.041779 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.042408 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.042511 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.046529 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:15.046535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:15.046539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:15.046576 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:15.050214 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.051590 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.051595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.051598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.051754 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.052220 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.052225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.052227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.052589 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:15.052848 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:15.053520 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 02:52:15.053759 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663}
I20260929 02:52:15.054839 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:15.033296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.4.197.188:35793)
0929 02:52:15.033476 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:15.033483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:15.035137 (+ 1654us) server_negotiation.cc:407] Connection header received
0929 02:52:15.035684 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:15.035695 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:15.035777 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:15.035848 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:15.038345 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.039514 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.041732 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.042464 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.042544 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.046505 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:15.046512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:15.046516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:15.046556 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:15.050195 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.051540 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.051546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.051549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.051730 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.052265 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.052270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.052273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.052656 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:15.052856 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:15.053811 (+ 955us) server_negotiation.cc:299] Negotiation successful
0929 02:52:15.054026 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:52:15.060173 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:15.032854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.4.197.190:33459)
0929 02:52:15.033039 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:15.033045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:15.042958 (+ 9913us) server_negotiation.cc:407] Connection header received
0929 02:52:15.043249 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:15.043255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:15.043340 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:15.043414 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:15.045851 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.047022 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.048768 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.049408 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.049512 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.053272 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:15.053281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:15.053287 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:15.053336 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:15.057076 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.058002 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.058006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.058009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.058140 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.058578 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.058583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.058586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.059033 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:15.059268 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:15.059589 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 02:52:15.059839 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 02:52:15.064678 18554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:52:15.064792 18554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:52:15.063298 18463 master_runner.cc:186] Error getting master registration for 127.4.197.190:33459: OK, instance_id { permanent_uuid: "8881c191fccc433989aee5664b380dff" instance_seqno: 1790650333987798 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:52:15.074457 18463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:52:15.125834 18382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } attempt: 4
Sep 29 02:52:15 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
I20260929 02:52:15.140988 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:15.126515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.4.197.188:35793)
0929 02:52:15.126688 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:15.126694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:15.126749 (+ 55us) server_negotiation.cc:407] Connection header received
0929 02:52:15.127048 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:15.127054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:15.127165 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:15.127254 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:15.128333 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.129403 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.131239 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.131715 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.131818 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.135572 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:15.135577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:15.135580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:15.135614 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:15.138376 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.139225 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.139229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.139232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.139372 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.139740 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.139743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.139745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.140022 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:15.140132 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:15.140545 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 02:52:15.140730 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:52:15.141175 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:15.126454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37662)
0929 02:52:15.126671 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:15.126692 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:15.126851 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:15.127445 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:15.127450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:15.127988 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:15.128201 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:15.128213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.129532 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.129539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:15.131078 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:15.131095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.131860 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.131867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:15.132082 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.132685 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:15.132714 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:15.135368 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:15.138513 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:15.138519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:15.138523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:15.139088 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:15.139471 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:15.139476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:15.139480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:15.139641 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:15.140120 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:15.140125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:15.140320 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 02:52:15.140461 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 02:52:15.140750 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:52:15.146345 18382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff: Bootstrap starting.
I20260929 02:52:15.149446 18382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:15.150517 18382 log.cc:824] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:15.153225 18382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff: No bootstrap required, opened a new log
I20260929 02:52:15.161404 18382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.161751 18382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:15.161864 18382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8881c191fccc433989aee5664b380dff, State: Initialized, Role: FOLLOWER
I20260929 02:52:15.162403 18382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.163350 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } } }
I20260929 02:52:15.163734 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:52:15.164521 18382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:52:15.181632 18568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:52:15.181803 18568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:52:15.174500 18315 master_runner.cc:186] Error getting master registration for 127.4.197.189:38123: OK, instance_id { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" instance_seqno: 1790650334463832 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:52:15.185890 18315 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:52:15.187307 18456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } attempt: 5
W20260929 02:52:15.188522 18484 tablet.cc:2369] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:52:15 dist-test-slave-t6c1 krb5kdc[18302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
I20260929 02:52:15.204491 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:15.188006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:35793 (local address 127.0.0.1:37666)
0929 02:52:15.188335 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:15.188354 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:15.188491 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:15.188918 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:15.188924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:15.189439 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:15.189651 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:15.189664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.191287 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.191294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:15.192721 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:15.192737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.193961 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.193968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:15.194215 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.194871 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:15.194896 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:15.197713 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:15.201700 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:15.201706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:15.201710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:15.202216 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:15.202611 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:15.202617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:15.202620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:15.202780 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:15.203488 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:15.203494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:15.203651 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 02:52:15.203808 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 02:52:15.204101 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260929 02:52:15.204845 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:15.188109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.4.197.188:35793)
0929 02:52:15.188314 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:15.188321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:15.188435 (+ 114us) server_negotiation.cc:407] Connection header received
0929 02:52:15.188654 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:15.188659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:15.188758 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:15.188866 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:15.189822 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.191051 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.192905 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:15.193629 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:15.193752 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:15.197939 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:15.197946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:15.197951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:15.197990 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:15.201565 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.202339 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.202344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.202348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.202490 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:15.202901 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:15.202907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:15.202910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:15.203322 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:15.203496 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:15.203929 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 02:52:15.204182 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:52:15.210495 18456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6: Bootstrap starting.
I20260929 02:52:15.212592 18543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:15.212888 18543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.214017 18557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:15.214262 18557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.214396 18543 leader_election.cc:288] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8881c191fccc433989aee5664b380dff (127.4.197.190:33459), b63494e451024f079447f87b6d1ff8e6 (127.4.197.189:38123)
I20260929 02:52:15.215075 18357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8881c191fccc433989aee5664b380dff" is_pre_election: true
I20260929 02:52:15.215554 18357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f8d5787066c4c3cb9eb8ec0b0af3f14 in term 0.
I20260929 02:52:15.215703 18557 leader_election.cc:288] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b63494e451024f079447f87b6d1ff8e6 (127.4.197.189:38123), 2f8d5787066c4c3cb9eb8ec0b0af3f14 (127.4.197.188:35793)
I20260929 02:52:15.216491 18482 leader_election.cc:302] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8d5787066c4c3cb9eb8ec0b0af3f14, 8881c191fccc433989aee5664b380dff; no voters:
I20260929 02:52:15.216656 18505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8881c191fccc433989aee5664b380dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" is_pre_election: true
I20260929 02:52:15.217130 18505 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8881c191fccc433989aee5664b380dff in term 0.
I20260929 02:52:15.217331 18543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:52:15.217478 18543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:52:15.217590 18543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:15.217900 18331 leader_election.cc:302] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8d5787066c4c3cb9eb8ec0b0af3f14, 8881c191fccc433989aee5664b380dff; no voters:
I20260929 02:52:15.218428 18557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:52:15.218569 18557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:52:15.218660 18557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:15.220708 18543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.221158 18456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:15.221788 18543 leader_election.cc:288] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [CANDIDATE]: Term 1 election: Requested vote from peers 8881c191fccc433989aee5664b380dff (127.4.197.190:33459), b63494e451024f079447f87b6d1ff8e6 (127.4.197.189:38123)
I20260929 02:52:15.221783 18557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.222043 18357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8881c191fccc433989aee5664b380dff"
I20260929 02:52:15.222324 18357 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f8d5787066c4c3cb9eb8ec0b0af3f14 in current term 1: Already voted for candidate 8881c191fccc433989aee5664b380dff in this term.
I20260929 02:52:15.223133 18505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8881c191fccc433989aee5664b380dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14"
I20260929 02:52:15.223398 18505 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8881c191fccc433989aee5664b380dff in current term 1: Already voted for candidate 2f8d5787066c4c3cb9eb8ec0b0af3f14 in this term.
I20260929 02:52:15.224242 18557 leader_election.cc:288] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [CANDIDATE]: Term 1 election: Requested vote from peers b63494e451024f079447f87b6d1ff8e6 (127.4.197.189:38123), 2f8d5787066c4c3cb9eb8ec0b0af3f14 (127.4.197.188:35793)
I20260929 02:52:15.224822 18456 log.cc:824] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6: Log is configured to *not* fsync() on all Append() calls
W20260929 02:52:15.228140 18336 tablet.cc:2369] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:52:15.234457 18456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6: No bootstrap required, opened a new log
I20260929 02:52:15.246212 18456 raft_consensus.cc:348] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.246719 18456 raft_consensus.cc:374] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:15.246855 18456 raft_consensus.cc:729] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63494e451024f079447f87b6d1ff8e6, State: Initialized, Role: FOLLOWER
I20260929 02:52:15.247696 18456 consensus_queue.cc:260] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.250362 18456 sys_catalog.cc:564] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:52:15.252502 18431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63494e451024f079447f87b6d1ff8e6" is_pre_election: true
I20260929 02:52:15.255402 18431 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f8d5787066c4c3cb9eb8ec0b0af3f14 in term 0.
I20260929 02:52:15.252660 18429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63494e451024f079447f87b6d1ff8e6"
I20260929 02:52:15.256568 18429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:15.252754 18430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8881c191fccc433989aee5664b380dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63494e451024f079447f87b6d1ff8e6" is_pre_election: true
I20260929 02:52:15.252843 18428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8881c191fccc433989aee5664b380dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63494e451024f079447f87b6d1ff8e6"
I20260929 02:52:15.260859 18332 leader_election.cc:302] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8881c191fccc433989aee5664b380dff; no voters: 2f8d5787066c4c3cb9eb8ec0b0af3f14, b63494e451024f079447f87b6d1ff8e6
I20260929 02:52:15.261381 18557 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 02:52:15.263072 18429 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f8d5787066c4c3cb9eb8ec0b0af3f14 in term 1.
I20260929 02:52:15.263608 18480 leader_election.cc:302] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f8d5787066c4c3cb9eb8ec0b0af3f14, b63494e451024f079447f87b6d1ff8e6; no voters: 8881c191fccc433989aee5664b380dff
I20260929 02:52:15.264176 18543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:52:15.264678 18543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [term 1 LEADER]: Becoming Leader. State: Replica: 2f8d5787066c4c3cb9eb8ec0b0af3f14, State: Running, Role: LEADER
I20260929 02:52:15.265285 18543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } }
I20260929 02:52:15.249396 18571 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } } }
I20260929 02:52:15.271166 18571 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:52:15.283972 18575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f8d5787066c4c3cb9eb8ec0b0af3f14. Latest consensus state: current_term: 1 leader_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } } }
I20260929 02:52:15.284415 18575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:15.288657 18579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:52:15.290908 18579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:52:15.300215 18429 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [term 1 FOLLOWER]: Refusing update from remote peer 2f8d5787066c4c3cb9eb8ec0b0af3f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:52:15.301204 18543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:52:15.312108 18357 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [term 1 FOLLOWER]: Refusing update from remote peer 2f8d5787066c4c3cb9eb8ec0b0af3f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:52:15.313028 18575 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:52:15.321691 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f8d5787066c4c3cb9eb8ec0b0af3f14. Latest consensus state: current_term: 1 leader_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } } }
I20260929 02:52:15.322067 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:52:15.324774 18581 mvcc.cc:205] Tried to move back new op lower bound from 7334503773377380352 to 7334503773251465216. Current Snapshot: MvccSnapshot[applied={T|T < 7334503773377380352}]
I20260929 02:52:15.325735 18587 mvcc.cc:205] Tried to move back new op lower bound from 7334503773377380352 to 7334503773251465216. Current Snapshot: MvccSnapshot[applied={T|T < 7334503773377380352}]
I20260929 02:52:15.328790 18575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } } }
I20260929 02:52:15.330509 18575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:15.331210 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } } }
I20260929 02:52:15.331475 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:52:15.333081 18571 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f8d5787066c4c3cb9eb8ec0b0af3f14. Latest consensus state: current_term: 1 leader_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } } }
I20260929 02:52:15.333384 18571 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:52:15.337433 18575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } } }
I20260929 02:52:15.337723 18575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:15.343503 18585 mvcc.cc:205] Tried to move back new op lower bound from 7334503773377380352 to 7334503773251465216. Current Snapshot: MvccSnapshot[applied={T|T < 7334503773377380352}]
I20260929 02:52:15.347478 18571 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8881c191fccc433989aee5664b380dff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33459 } } peers { permanent_uuid: "b63494e451024f079447f87b6d1ff8e6" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 38123 } } peers { permanent_uuid: "2f8d5787066c4c3cb9eb8ec0b0af3f14" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 35793 } } }
I20260929 02:52:15.347748 18571 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:52:15.349829 18579 catalog_manager.cc:1347] Generated new cluster ID: 29320dbb24ba4a3393f6429104f35a52
I20260929 02:52:15.349936 18579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:52:15.378439 18592 catalog_manager.cc:1259] Loaded cluster ID: 29320dbb24ba4a3393f6429104f35a52
I20260929 02:52:15.378711 18592 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6: loading cluster ID for follower catalog manager: success
W20260929 02:52:15.380221 18592 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260929 02:52:15.380481 18592 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:52:15.411072 18579 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:52:15.413192 18579 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:52:15.446488 18579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f8d5787066c4c3cb9eb8ec0b0af3f14: Generated new TSK 0
I20260929 02:52:15.447734 18579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:52:16.183890 18568 catalog_manager.cc:1259] Loaded cluster ID: 29320dbb24ba4a3393f6429104f35a52
I20260929 02:52:16.183997 18568 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff: loading cluster ID for follower catalog manager: success
I20260929 02:52:16.186681 18568 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff: acquiring CA information for follower catalog manager: success
I20260929 02:52:16.188700 18568 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8881c191fccc433989aee5664b380dff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:52:16.383847 18592 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6: acquiring CA information for follower catalog manager: success
I20260929 02:52:16.385785 18592 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b63494e451024f079447f87b6d1ff8e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:52:16.603868 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 18315
I20260929 02:52:16.605034 18377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:16.716938 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 18315
I20260929 02:52:16.734607 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 18389
I20260929 02:52:16.735864 18451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:16.842440 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 18389
I20260929 02:52:16.859038 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 18463
I20260929 02:52:16.860049 18525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:16.967721 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 18463
2026-09-29T02:52:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4617 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:52:17 dist-test-slave-t6c1 krb5kdc[18604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:52:17 dist-test-slave-t6c1 krb5kdc[18604](info): set up 2 sockets
Sep 29 02:52:17 dist-test-slave-t6c1 krb5kdc[18604](info): commencing operation
krb5kdc: starting...
W20260929 02:52:17.979491 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:52:18 dist-test-slave-t6c1 krb5kdc[18604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:52:18.196627 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:35007
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:35007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:18.517016 18617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:18.517274 18617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:18.517390 18617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:18.524338 18617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:52:18.524456 18617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:18.524526 18617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:18.524601 18617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:52:18.524643 18617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:52:18.534667 18617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:35007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:18.536082 18617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:18.537499 18617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:18.548681 18622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:18.548717 18623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:18.548840 18625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:18.549629 18617 server_base.cc:1034] running on GCE node
I20260929 02:52:18.550374 18617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:18.551730 18617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:18.552937 18617 hybrid_clock.cc:648] HybridClock initialized: now 1790650338552885 us; error 37 us; skew 500 ppm
I20260929 02:52:18.553251 18617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:18 dist-test-slave-t6c1 krb5kdc[18604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650338, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:18.556174 18617 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:52:18.557494 18617 webserver.cc:460] Webserver started at http://127.4.197.190:40115/ using document root <none> and password file <none>
I20260929 02:52:18.558161 18617 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:18.558259 18617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:18.558514 18617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:18.561421 18617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "563db8cb04d141adb34c15e639de141c"
format_stamp: "Formatted at 2026-09-29 02:52:18 on dist-test-slave-t6c1"
server_key: "50f0e5da8f51c0fa975f66c2b8fe16d5"
server_key_iv: "11109acca08def0e0fe9a0b622ec165a"
server_key_version: "encryptionkey@0"
I20260929 02:52:18.562163 18617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "563db8cb04d141adb34c15e639de141c"
format_stamp: "Formatted at 2026-09-29 02:52:18 on dist-test-slave-t6c1"
server_key: "50f0e5da8f51c0fa975f66c2b8fe16d5"
server_key_iv: "11109acca08def0e0fe9a0b622ec165a"
server_key_version: "encryptionkey@0"
I20260929 02:52:18.567166 18617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:52:18.570493 18632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:18.571898 18617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:52:18.572053 18617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "563db8cb04d141adb34c15e639de141c"
format_stamp: "Formatted at 2026-09-29 02:52:18 on dist-test-slave-t6c1"
server_key: "50f0e5da8f51c0fa975f66c2b8fe16d5"
server_key_iv: "11109acca08def0e0fe9a0b622ec165a"
server_key_version: "encryptionkey@0"
I20260929 02:52:18.572240 18617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:18.595875 18617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:18.600306 18617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:18.600575 18617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:18.618456 18617 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:35007
I20260929 02:52:18.618476 18683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:35007 every 8 connection(s)
I20260929 02:52:18.619927 18617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:52:18.623905 18684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:18.628443 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 18617
I20260929 02:52:18.628680 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:52:18.629513 4886 external_mini_cluster.cc:1428] Setting key 7adacff0a57bead0bd754ce892d43cff
I20260929 02:52:18.635710 18684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c: Bootstrap starting.
I20260929 02:52:18.639307 18684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:18.640489 18684 log.cc:824] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:18.644047 18684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c: No bootstrap required, opened a new log
Sep 29 02:52:18 dist-test-slave-t6c1 krb5kdc[18604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:18.652192 18684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563db8cb04d141adb34c15e639de141c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35007 } }
I20260929 02:52:18.652693 18684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:18.652819 18684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563db8cb04d141adb34c15e639de141c, State: Initialized, Role: FOLLOWER
I20260929 02:52:18.653460 18684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563db8cb04d141adb34c15e639de141c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35007 } }
I20260929 02:52:18.653705 18684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:52:18.653846 18684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:52:18.653950 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:18.631486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.4.197.190:35007)
0929 02:52:18.632265 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:18.632278 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:18.632319 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:52:18.634231 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:18.634256 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:18.634654 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:18.635024 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:18.636824 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:18.638176 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:18.639935 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:18.640660 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:18.640807 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:18.645930 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:18.645937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:18.645955 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:18.646001 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:18.649228 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:18.649981 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:18.649987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:18.649995 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:18.650225 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:18.650691 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:18.650695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:18.650697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:18.651366 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:18.651658 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:18.652119 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 02:52:18.652591 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":365,"threads_started":1}
I20260929 02:52:18.654019 18684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:18.656651 18684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563db8cb04d141adb34c15e639de141c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35007 } }
I20260929 02:52:18.657300 18684 leader_election.cc:302] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 563db8cb04d141adb34c15e639de141c; no voters:
I20260929 02:52:18.657917 18684 leader_election.cc:288] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:52:18.658066 18689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:52:18.658416 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:52:18.659261 18689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [term 1 LEADER]: Becoming Leader. State: Replica: 563db8cb04d141adb34c15e639de141c, State: Running, Role: LEADER
I20260929 02:52:18.659878 18689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563db8cb04d141adb34c15e639de141c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35007 } }
I20260929 02:52:18.660251 18684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:52:18.667542 18691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 563db8cb04d141adb34c15e639de141c. Latest consensus state: current_term: 1 leader_uuid: "563db8cb04d141adb34c15e639de141c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563db8cb04d141adb34c15e639de141c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35007 } } }
I20260929 02:52:18.667979 18691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:18.667601 18690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "563db8cb04d141adb34c15e639de141c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563db8cb04d141adb34c15e639de141c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35007 } } }
I20260929 02:52:18.668145 18690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:18.671499 18698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:52:18.675611 18698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:52:18.683933 18698 catalog_manager.cc:1347] Generated new cluster ID: e9eb88d6d08f458c9faebaabe050ac81
I20260929 02:52:18.684044 18698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:52:18.707239 18698 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:52:18.708546 18698 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:52:18.715553 18698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 563db8cb04d141adb34c15e639de141c: Generated new TSK 0
I20260929 02:52:18.716300 18698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:52:18.771693 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 18617
I20260929 02:52:18.772915 18679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:18.869382 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 18617
2026-09-29T02:52:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1898 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:52:18 dist-test-slave-t6c1 krb5kdc[18712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:52:18 dist-test-slave-t6c1 krb5kdc[18712](info): set up 2 sockets
Sep 29 02:52:18 dist-test-slave-t6c1 krb5kdc[18712](info): commencing operation
krb5kdc: starting...
W20260929 02:52:19.867354 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:52:20 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:52:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:52:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:52:20.082043 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36783
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:36783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:20.406463 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:20.406749 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:20.406888 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:20.413893 18725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:52:20.414006 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:20.414080 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:20.414153 18725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:52:20.414211 18725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:52:20.423661 18725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:36783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:20.424999 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:20.426327 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:20.437951 18731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:20.437924 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:20.438227 18730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:20.438963 18725 server_base.cc:1034] running on GCE node
I20260929 02:52:20.439793 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:20.441284 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:20.442484 18725 hybrid_clock.cc:648] HybridClock initialized: now 1790650340442432 us; error 38 us; skew 500 ppm
I20260929 02:52:20.442862 18725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:20 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650340, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:20.445992 18725 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:52:20.447590 18725 webserver.cc:460] Webserver started at http://127.4.197.190:41049/ using document root <none> and password file <none>
I20260929 02:52:20.448271 18725 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:20.448388 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:20.448676 18725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:20.451704 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "d73bb450cfbf46d0a18ac7638323a0d2"
format_stamp: "Formatted at 2026-09-29 02:52:20 on dist-test-slave-t6c1"
server_key: "f82ef543c20c094230cf0f40f353127f"
server_key_iv: "5ccf4a4dc22ecc6a39895dec67a0d8b1"
server_key_version: "encryptionkey@0"
I20260929 02:52:20.452445 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "d73bb450cfbf46d0a18ac7638323a0d2"
format_stamp: "Formatted at 2026-09-29 02:52:20 on dist-test-slave-t6c1"
server_key: "f82ef543c20c094230cf0f40f353127f"
server_key_iv: "5ccf4a4dc22ecc6a39895dec67a0d8b1"
server_key_version: "encryptionkey@0"
I20260929 02:52:20.457698 18725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:52:20.461401 18740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:20.462896 18725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 02:52:20.463141 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "d73bb450cfbf46d0a18ac7638323a0d2"
format_stamp: "Formatted at 2026-09-29 02:52:20 on dist-test-slave-t6c1"
server_key: "f82ef543c20c094230cf0f40f353127f"
server_key_iv: "5ccf4a4dc22ecc6a39895dec67a0d8b1"
server_key_version: "encryptionkey@0"
I20260929 02:52:20.463336 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:20.479535 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:20.484210 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:20.484534 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:20.502123 18725 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:36783
I20260929 02:52:20.502131 18791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:36783 every 8 connection(s)
I20260929 02:52:20.503698 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:52:20.507469 18792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:20.512071 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 18725
I20260929 02:52:20.512295 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:52:20.513152 4886 external_mini_cluster.cc:1428] Setting key d204df69e82623681ae5256ad9793855
I20260929 02:52:20.518671 18792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Bootstrap starting.
I20260929 02:52:20.522243 18792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:20.523573 18792 log.cc:824] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:20.526662 18792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: No bootstrap required, opened a new log
Sep 29 02:52:20 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:20.535877 18792 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:20.536284 18792 raft_consensus.cc:374] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:20.536386 18792 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73bb450cfbf46d0a18ac7638323a0d2, State: Initialized, Role: FOLLOWER
I20260929 02:52:20.537073 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:20.515215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.4.197.190:36783)
0929 02:52:20.516021 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:20.516036 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:20.516078 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:52:20.517784 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:20.517807 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:20.518196 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:20.518525 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:20.520220 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:20.521521 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:20.523340 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:20.524139 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:20.524387 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:20.528720 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:20.528728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:20.528748 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:20.528787 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:20.531942 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:20.532780 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:20.532790 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:20.532796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:20.532967 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:20.533366 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:20.533369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:20.533371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:20.533934 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:20.534323 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:20.535442 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0929 02:52:20.535822 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":368,"threads_started":1}
I20260929 02:52:20.536999 18792 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:20.537263 18792 raft_consensus.cc:388] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:52:20.537408 18792 raft_consensus.cc:482] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:52:20.537587 18792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:20.539981 18792 raft_consensus.cc:504] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:20.540536 18792 leader_election.cc:302] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73bb450cfbf46d0a18ac7638323a0d2; no voters:
I20260929 02:52:20.541145 18792 leader_election.cc:288] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:52:20.541308 18797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:52:20.541936 18797 raft_consensus.cc:686] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 1 LEADER]: Becoming Leader. State: Replica: d73bb450cfbf46d0a18ac7638323a0d2, State: Running, Role: LEADER
I20260929 02:52:20.542464 18797 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:20.543229 18792 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:52:20.545266 18798 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } } }
I20260929 02:52:20.545648 18798 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:20.546787 18799 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73bb450cfbf46d0a18ac7638323a0d2. Latest consensus state: current_term: 1 leader_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } } }
I20260929 02:52:20.547143 18799 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:20.550172 18802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:52:20.555397 18802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:52:20.563954 18802 catalog_manager.cc:1347] Generated new cluster ID: f9ba519101284f7087a7c1b82d492ba1
I20260929 02:52:20.564065 18802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:52:20.583583 18802 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:52:20.585646 18802 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:52:20.598704 18802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Generated new TSK 0
I20260929 02:52:20.599823 18802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:52:20.657051 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36783
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:20.992796 18818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:20.993101 18818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:20.993248 18818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:21.000458 18818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:21.000572 18818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:21.000720 18818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:52:21.010306 18818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:21.011601 18818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:21.012929 18818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:21.025338 18823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:21.025482 18824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:21.025321 18826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:21.026470 18818 server_base.cc:1034] running on GCE node
I20260929 02:52:21.027144 18818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:21.028127 18818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:21.029341 18818 hybrid_clock.cc:648] HybridClock initialized: now 1790650341029298 us; error 51 us; skew 500 ppm
I20260929 02:52:21.029650 18818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:21 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650341, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:21.032786 18818 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:52:21.034332 18818 webserver.cc:460] Webserver started at http://127.4.197.129:40871/ using document root <none> and password file <none>
I20260929 02:52:21.035060 18818 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:21.035171 18818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:21.035511 18818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:21.038460 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "3e5bc99fdc974c31b6ab0b6c28035018"
format_stamp: "Formatted at 2026-09-29 02:52:21 on dist-test-slave-t6c1"
server_key: "8aa0f29d70e1bbc532df8a90db8b9166"
server_key_iv: "aee05fb425ba48be10d841abc519fdeb"
server_key_version: "encryptionkey@0"
I20260929 02:52:21.039239 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "3e5bc99fdc974c31b6ab0b6c28035018"
format_stamp: "Formatted at 2026-09-29 02:52:21 on dist-test-slave-t6c1"
server_key: "8aa0f29d70e1bbc532df8a90db8b9166"
server_key_iv: "aee05fb425ba48be10d841abc519fdeb"
server_key_version: "encryptionkey@0"
I20260929 02:52:21.044690 18818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:52:21.048542 18833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:21.050073 18818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:52:21.050236 18818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "3e5bc99fdc974c31b6ab0b6c28035018"
format_stamp: "Formatted at 2026-09-29 02:52:21 on dist-test-slave-t6c1"
server_key: "8aa0f29d70e1bbc532df8a90db8b9166"
server_key_iv: "aee05fb425ba48be10d841abc519fdeb"
server_key_version: "encryptionkey@0"
I20260929 02:52:21.050423 18818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:21.069852 18818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:21.073951 18818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:21.074198 18818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:21.075126 18818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:21.076455 18818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:21.076526 18818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:21.076623 18818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:21.076665 18818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:21.106868 18818 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:43297
I20260929 02:52:21.106910 18945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:43297 every 8 connection(s)
I20260929 02:52:21.108104 18818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:52:21.111168 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 18818
I20260929 02:52:21.111349 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:52:21.111941 4886 external_mini_cluster.cc:1428] Setting key a08ad8b75acb91ef18f5a0baf1a1bb4c
Sep 29 02:52:21 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650341, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:21.128343 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:21.110478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:39001 (local address 127.4.197.190:36783)
0929 02:52:21.110910 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:21.110944 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:21.113942 (+ 2998us) server_negotiation.cc:407] Connection header received
0929 02:52:21.113991 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:21.113995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:21.114054 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:21.114129 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:21.116041 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:21.116744 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:21.118808 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:21.119244 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:21.119340 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:21.123059 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:21.123064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:21.123067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:21.123097 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:21.125503 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:21.126164 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:21.126168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:21.126171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:21.126293 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:21.126682 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:21.126686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:21.126688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:21.126976 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:21.127106 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:21.127754 (+ 648us) server_negotiation.cc:299] Negotiation successful
0929 02:52:21.127976 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":192,"threads_started":1}
I20260929 02:52:21.129212 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:21.111066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:39001)
0929 02:52:21.111868 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:21.111917 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:21.113691 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:21.114436 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:21.114449 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:21.115112 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:21.115852 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:21.115875 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:21.116895 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:21.116900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:21.118681 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:21.118695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:21.119446 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:21.119451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:21.119701 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:21.120346 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:21.120376 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:21.122844 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:21.125625 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:21.125633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:21.125649 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:21.126043 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:21.126388 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:21.126394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:21.126398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:21.126572 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:21.127087 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:21.127094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:21.127404 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 02:52:21.127626 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 02:52:21.128122 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":310,"threads_started":1}
I20260929 02:52:21.130658 18946 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36783
I20260929 02:52:21.130913 18946 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:21.131611 18946 heartbeater.cc:511] Master 127.4.197.190:36783 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:52:21.133608 18757 ts_manager.cc:194] Registered new tserver with Master: 3e5bc99fdc974c31b6ab0b6c28035018 (127.4.197.129:43297)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
I20260929 02:52:21.135179 18757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:39001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:52:21.158300 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36783
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:21.474748 18952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:21.475054 18952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:21.475154 18952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:21.482380 18952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:21.482511 18952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:21.482671 18952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:52:21.492676 18952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:21.493999 18952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:21.495332 18952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:21.508339 18957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:21.508255 18958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:21.508455 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:21.509275 18952 server_base.cc:1034] running on GCE node
I20260929 02:52:21.509833 18952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:21.510723 18952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:21.511924 18952 hybrid_clock.cc:648] HybridClock initialized: now 1790650341511866 us; error 47 us; skew 500 ppm
I20260929 02:52:21.512220 18952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:21 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650341, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:21.515105 18952 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:52:21.516476 18952 webserver.cc:460] Webserver started at http://127.4.197.130:37633/ using document root <none> and password file <none>
I20260929 02:52:21.517093 18952 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:21.517207 18952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:21.517467 18952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:21.520260 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "e8112e9d22c74879b18abe13e333f5f6"
format_stamp: "Formatted at 2026-09-29 02:52:21 on dist-test-slave-t6c1"
server_key: "dfebef9a0d867bb83f2a78b2c0c02f04"
server_key_iv: "7402b3912d0122611af8c369865b7834"
server_key_version: "encryptionkey@0"
I20260929 02:52:21.520924 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "e8112e9d22c74879b18abe13e333f5f6"
format_stamp: "Formatted at 2026-09-29 02:52:21 on dist-test-slave-t6c1"
server_key: "dfebef9a0d867bb83f2a78b2c0c02f04"
server_key_iv: "7402b3912d0122611af8c369865b7834"
server_key_version: "encryptionkey@0"
I20260929 02:52:21.525774 18952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:52:21.529078 18967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:21.530416 18952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:52:21.530561 18952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "e8112e9d22c74879b18abe13e333f5f6"
format_stamp: "Formatted at 2026-09-29 02:52:21 on dist-test-slave-t6c1"
server_key: "dfebef9a0d867bb83f2a78b2c0c02f04"
server_key_iv: "7402b3912d0122611af8c369865b7834"
server_key_version: "encryptionkey@0"
I20260929 02:52:21.530761 18952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:21.564389 18952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:21.568749 18952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:21.569008 18952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:21.569952 18952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:21.571350 18952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:21.571424 18952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:21.571545 18952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:21.571593 18952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:21.600535 18952 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:46797
I20260929 02:52:21.600574 19079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:46797 every 8 connection(s)
I20260929 02:52:21.601728 18952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:52:21.609584 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 18952
I20260929 02:52:21.609789 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:52:21.610229 4886 external_mini_cluster.cc:1428] Setting key f5c1c5b027ac519215005298eaea052e
Sep 29 02:52:21 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650341, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:21.620332 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:21.604075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:50087 (local address 127.4.197.190:36783)
0929 02:52:21.604272 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:21.604276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:21.605380 (+ 1104us) server_negotiation.cc:407] Connection header received
0929 02:52:21.607175 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:21.607181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:21.607264 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:21.607348 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:21.608775 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:21.609600 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:21.611318 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:21.611920 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:21.612028 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:21.615219 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:21.615224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:21.615227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:21.615256 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:21.617604 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:21.618235 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:21.618239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:21.618241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:21.618359 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:21.618743 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:21.618746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:21.618748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:21.619026 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:21.619162 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:21.619765 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 02:52:21.619993 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:52:21.621308 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:21.604581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:50087)
0929 02:52:21.605221 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:21.605261 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:21.606879 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:21.607584 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:21.607591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:21.608078 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:21.608607 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:21.608635 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:21.609807 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:21.609811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:21.611196 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:21.611209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:21.612126 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:21.612135 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:21.612346 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:21.612979 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:21.613004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:21.615021 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:21.617726 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:21.617734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:21.617750 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:21.618120 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:21.618454 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:21.618459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:21.618463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:21.618636 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:21.619136 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:21.619143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:21.619435 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 02:52:21.619662 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 02:52:21.620133 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":251,"threads_started":1}
I20260929 02:52:21.622633 19080 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36783
I20260929 02:52:21.622962 19080 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:21.623754 19080 heartbeater.cc:511] Master 127.4.197.190:36783 requested a full tablet report, sending...
I20260929 02:52:21.625388 18757 ts_manager.cc:194] Registered new tserver with Master: e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797)
I20260929 02:52:21.626451 18757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:50087
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:52:21.656358 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36783
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:21.973271 19085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:21.973591 19085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:21.973732 19085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:21.981072 19085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:21.981194 19085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:21.981351 19085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:52:21.991248 19085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:21.992486 19085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:21.993731 19085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:22.005911 19090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:22.006006 19093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:22.006048 19091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:22.006875 19085 server_base.cc:1034] running on GCE node
I20260929 02:52:22.007445 19085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:22.008347 19085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:22.009529 19085 hybrid_clock.cc:648] HybridClock initialized: now 1790650342009489 us; error 37 us; skew 500 ppm
I20260929 02:52:22.009814 19085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:22 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650342, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:22.012826 19085 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:52:22.014235 19085 webserver.cc:460] Webserver started at http://127.4.197.131:39745/ using document root <none> and password file <none>
I20260929 02:52:22.014911 19085 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:22.015094 19085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:22.015424 19085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:22.018342 19085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "733e27ba999245dba0b982eeb1a361a7"
format_stamp: "Formatted at 2026-09-29 02:52:22 on dist-test-slave-t6c1"
server_key: "8ef1fdfa2364c2b1083934ea72d69676"
server_key_iv: "dbf890c84228ba136d07918be70f0929"
server_key_version: "encryptionkey@0"
I20260929 02:52:22.019217 19085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "733e27ba999245dba0b982eeb1a361a7"
format_stamp: "Formatted at 2026-09-29 02:52:22 on dist-test-slave-t6c1"
server_key: "8ef1fdfa2364c2b1083934ea72d69676"
server_key_iv: "dbf890c84228ba136d07918be70f0929"
server_key_version: "encryptionkey@0"
I20260929 02:52:22.024555 19085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:52:22.027997 19100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:22.029438 19085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:52:22.029580 19085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "733e27ba999245dba0b982eeb1a361a7"
format_stamp: "Formatted at 2026-09-29 02:52:22 on dist-test-slave-t6c1"
server_key: "8ef1fdfa2364c2b1083934ea72d69676"
server_key_iv: "dbf890c84228ba136d07918be70f0929"
server_key_version: "encryptionkey@0"
I20260929 02:52:22.029762 19085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:22.067914 19085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:22.071993 19085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:22.072242 19085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:22.073187 19085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:22.074507 19085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:22.074582 19085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:22.074689 19085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:22.074733 19085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:22.103740 19085 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:33919
I20260929 02:52:22.103762 19212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:33919 every 8 connection(s)
I20260929 02:52:22.104964 19085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:52:22.107061 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 19085
I20260929 02:52:22.107255 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:52:22.107812 4886 external_mini_cluster.cc:1428] Setting key a4dbd7d0094ee89b22131ec058fcbc5c
Sep 29 02:52:22 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650342, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:22.127401 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.112206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:38789 (local address 127.4.197.190:36783)
0929 02:52:22.112397 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.112402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.113513 (+ 1111us) server_negotiation.cc:407] Connection header received
0929 02:52:22.115190 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.115194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.115257 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.115328 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:22.116588 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.117347 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.118743 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.119155 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.119243 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.122503 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:22.122508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:22.122514 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:22.122548 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:22.124899 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:22.125505 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:22.125509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:22.125512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:22.125667 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:22.126009 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:22.126012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:22.126014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:22.126258 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:22.126368 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.126988 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.127156 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:52:22.128364 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.112719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:38789)
0929 02:52:22.113363 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:22.113405 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:22.114956 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:22.115484 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:22.115491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:22.115945 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:22.116467 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.116483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.117464 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.117467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.118641 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.118652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.119315 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.119320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.119506 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.120051 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:22.120076 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:22.122330 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:22.125012 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:22.125018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:22.125029 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:22.125422 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:22.125750 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:22.125753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:22.125755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:22.125928 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:22.126349 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:22.126355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:22.126653 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 02:52:22.126884 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 02:52:22.127290 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":243,"threads_started":1}
I20260929 02:52:22.129617 19213 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36783
I20260929 02:52:22.129887 19213 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:22.130543 19213 heartbeater.cc:511] Master 127.4.197.190:36783 requested a full tablet report, sending...
I20260929 02:52:22.131918 18757 ts_manager.cc:194] Registered new tserver with Master: 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919)
I20260929 02:52:22.132717 18757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:38789
I20260929 02:52:22.135138 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:52:22.137893 18946 heartbeater.cc:503] Master 127.4.197.190:36783 was elected leader, sending a full tablet report...
I20260929 02:52:22.152621 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.139881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.4.197.190:36783)
0929 02:52:22.140064 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.140070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.140423 (+ 353us) server_negotiation.cc:407] Connection header received
0929 02:52:22.140617 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.140621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.140686 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.140766 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:22.142175 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.142947 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.144331 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.144806 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.144918 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.147373 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:22.147378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:22.147381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:22.147417 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:22.149828 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:22.150589 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:22.150593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:22.150596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:22.150735 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:22.151158 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:22.151162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:22.151164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:22.151483 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:22.151751 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.152079 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.152305 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:52:22.156724 18757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:52:22.163502 18757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:52:22.199790 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.175916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.4.197.129:43297)
0929 02:52:22.191281 (+ 15365us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.191292 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.191327 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:52:22.191448 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.191458 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.191761 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.192955 (+ 1194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:22.193210 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.194783 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.197137 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.198659 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.198803 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.199068 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.199315 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.199482 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15159,"thread_start_us":16012,"threads_started":1}
I20260929 02:52:22.199884 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.175724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.0.0.1:50814)
0929 02:52:22.177025 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:22.177061 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:22.177235 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:22.192271 (+ 15036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:22.192280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:22.192324 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:22.192732 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.192745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.194977 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.194982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.196992 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.197004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.198855 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.198862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.199091 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.199140 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 02:52:22.199270 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 02:52:22.199459 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":1,"thread_start_us":919,"threads_started":1}
I20260929 02:52:22.200843 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.176324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:33919 (local address 127.0.0.1:39154)
0929 02:52:22.191384 (+ 15060us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:22.191406 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:22.191529 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:22.193951 (+ 2422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:22.193955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:22.193979 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:22.194393 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.194407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.196328 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.196334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.198456 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.198470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.200134 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.200139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.200282 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.200324 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:52:22.200387 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 02:52:22.200500 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14905,"mutex_wait_us":42,"thread_start_us":18020,"threads_started":1}
I20260929 02:52:22.201810 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.176965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.4.197.131:33919)
0929 02:52:22.193371 (+ 16406us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.193380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.193415 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:52:22.193509 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.193517 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.193769 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.193955 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:22.194623 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.196194 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.198616 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.199919 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.201160 (+ 1241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.201388 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.201452 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.201563 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16216,"thread_start_us":16045,"threads_started":1}
I20260929 02:52:22.202783 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.177416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:46797 (local address 127.0.0.1:57258)
0929 02:52:22.194645 (+ 17229us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:22.194663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:22.194778 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:22.195306 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:22.195310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:22.195335 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:22.195713 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.195723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.197367 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.197372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.199342 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.199355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.202142 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.202148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.202266 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.202287 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 02:52:22.202362 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 02:52:22.202477 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17096,"thread_start_us":14200,"threads_started":1}
I20260929 02:52:22.202881 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.177759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.4.197.130:46797)
0929 02:52:22.192195 (+ 14436us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.192204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.194774 (+ 2570us) server_negotiation.cc:407] Connection header received
0929 02:52:22.194938 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.194947 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.195128 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.195295 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:22.195874 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.197200 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.199491 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.200849 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.200946 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.201186 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.202420 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.202570 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14257,"thread_start_us":14434,"threads_started":1}
I20260929 02:52:22.204036 18881 tablet_service.cc:1467] Processing CreateTablet for tablet 38a091509dfd4c3d8216f0e86ba3059a (DEFAULT_TABLE table=test-table [id=8ad0a83b7a6a4a329068c8b138a898d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:22.205541 18881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a091509dfd4c3d8216f0e86ba3059a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:22.205296 19148 tablet_service.cc:1467] Processing CreateTablet for tablet 38a091509dfd4c3d8216f0e86ba3059a (DEFAULT_TABLE table=test-table [id=8ad0a83b7a6a4a329068c8b138a898d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:22.205822 19015 tablet_service.cc:1467] Processing CreateTablet for tablet 38a091509dfd4c3d8216f0e86ba3059a (DEFAULT_TABLE table=test-table [id=8ad0a83b7a6a4a329068c8b138a898d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:22.206610 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a091509dfd4c3d8216f0e86ba3059a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:22.206808 19015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a091509dfd4c3d8216f0e86ba3059a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:22.216637 19231 tablet_bootstrap.cc:492] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: Bootstrap starting.
I20260929 02:52:22.219249 19232 tablet_bootstrap.cc:492] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: Bootstrap starting.
I20260929 02:52:22.219976 19231 tablet_bootstrap.cc:654] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:22.220629 19233 tablet_bootstrap.cc:492] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: Bootstrap starting.
I20260929 02:52:22.221457 19231 log.cc:824] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:22.222486 19232 tablet_bootstrap.cc:654] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:22.223541 19232 log.cc:824] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:22.223655 19233 tablet_bootstrap.cc:654] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:22.224700 19233 log.cc:824] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:22.225149 19231 tablet_bootstrap.cc:492] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: No bootstrap required, opened a new log
I20260929 02:52:22.225497 19231 ts_tablet_manager.cc:1397] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:52:22.227313 19232 tablet_bootstrap.cc:492] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: No bootstrap required, opened a new log
I20260929 02:52:22.227515 19233 tablet_bootstrap.cc:492] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: No bootstrap required, opened a new log
I20260929 02:52:22.227540 19232 ts_tablet_manager.cc:1397] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:52:22.227757 19233 ts_tablet_manager.cc:1397] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 02:52:22.234639 19232 raft_consensus.cc:348] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } } peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } }
I20260929 02:52:22.235090 19232 raft_consensus.cc:374] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:22.235198 19232 raft_consensus.cc:729] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5bc99fdc974c31b6ab0b6c28035018, State: Initialized, Role: FOLLOWER
I20260929 02:52:22.235651 19231 raft_consensus.cc:348] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } } peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } }
I20260929 02:52:22.235926 19232 consensus_queue.cc:260] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } } peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } }
I20260929 02:52:22.236177 19231 raft_consensus.cc:374] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:22.236311 19231 raft_consensus.cc:729] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8112e9d22c74879b18abe13e333f5f6, State: Initialized, Role: FOLLOWER
I20260929 02:52:22.236961 19231 consensus_queue.cc:260] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } } peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } }
I20260929 02:52:22.237846 19233 raft_consensus.cc:348] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } } peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } }
I20260929 02:52:22.238246 19233 raft_consensus.cc:374] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:22.238247 19080 heartbeater.cc:503] Master 127.4.197.190:36783 was elected leader, sending a full tablet report...
I20260929 02:52:22.238344 19233 raft_consensus.cc:729] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733e27ba999245dba0b982eeb1a361a7, State: Initialized, Role: FOLLOWER
I20260929 02:52:22.238902 19233 consensus_queue.cc:260] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } } peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } }
I20260929 02:52:22.240327 19213 heartbeater.cc:503] Master 127.4.197.190:36783 was elected leader, sending a full tablet report...
I20260929 02:52:22.241890 19233 ts_tablet_manager.cc:1428] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260929 02:52:22.243460 19231 ts_tablet_manager.cc:1428] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: Time spent starting tablet: real 0.018s user 0.012s sys 0.001s
I20260929 02:52:22.244179 19232 ts_tablet_manager.cc:1428] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
W20260929 02:52:22.353580 19081 tablet.cc:2369] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:52:22.356617 19214 tablet.cc:2369] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:52:22.360183 18947 tablet.cc:2369] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:52:22.498942 19238 raft_consensus.cc:482] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:22.499328 19238 raft_consensus.cc:504] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } } peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } }
I20260929 02:52:22.501248 19238 leader_election.cc:288] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e5bc99fdc974c31b6ab0b6c28035018 (127.4.197.129:43297), 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919)
I20260929 02:52:22.509894 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.501861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:39623 (local address 127.4.197.129:43297)
0929 02:52:22.502126 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.502132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.503334 (+ 1202us) server_negotiation.cc:407] Connection header received
0929 02:52:22.503667 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.503675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.503763 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.503863 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:22.504469 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.505358 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.507676 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.508935 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.509096 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.509171 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.509515 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.509653 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 02:52:22.510011 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.501530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.4.197.130:39623)
0929 02:52:22.503192 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:22.503214 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:22.503474 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:22.503875 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:22.503880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:22.503904 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:22.504298 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.504309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.505489 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.505494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.507511 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.507525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.509154 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.509161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.509350 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.509384 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:52:22.509484 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:52:22.509617 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1512,"mutex_wait_us":41,"thread_start_us":1785,"threads_started":1}
I20260929 02:52:22.510006 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.502432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:54673 (local address 127.4.197.131:33919)
0929 02:52:22.502645 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.502651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.503784 (+ 1133us) server_negotiation.cc:407] Connection header received
0929 02:52:22.503981 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.503987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.504065 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.504119 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:22.504922 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.506028 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.508186 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.508956 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.509104 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.509178 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.509650 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.509789 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:52:22.510625 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.501530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:33919 (local address 127.4.197.130:54673)
0929 02:52:22.503659 (+ 2129us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:22.503682 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:22.503808 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:22.504285 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:22.504288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:22.504316 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:22.504730 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.504743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.506186 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.506191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.508034 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:22.508044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.509329 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.509335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:22.509494 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.509520 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 02:52:22.510171 (+ 651us) client_negotiation.cc:240] Negotiation successful
0929 02:52:22.510307 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2004,"mutex_wait_us":66,"thread_start_us":1283,"threads_started":1}
I20260929 02:52:22.511020 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a091509dfd4c3d8216f0e86ba3059a" candidate_uuid: "e8112e9d22c74879b18abe13e333f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" is_pre_election: true
I20260929 02:52:22.511580 19168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a091509dfd4c3d8216f0e86ba3059a" candidate_uuid: "e8112e9d22c74879b18abe13e333f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733e27ba999245dba0b982eeb1a361a7" is_pre_election: true
I20260929 02:52:22.512123 19168 raft_consensus.cc:2454] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8112e9d22c74879b18abe13e333f5f6 in term 0.
I20260929 02:52:22.512534 18901 raft_consensus.cc:2454] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8112e9d22c74879b18abe13e333f5f6 in term 0.
I20260929 02:52:22.512804 18968 leader_election.cc:302] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 733e27ba999245dba0b982eeb1a361a7, e8112e9d22c74879b18abe13e333f5f6; no voters:
I20260929 02:52:22.513221 19238 raft_consensus.cc:2781] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:52:22.513370 19238 raft_consensus.cc:482] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:52:22.513453 19238 raft_consensus.cc:3037] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:22.516490 19238 raft_consensus.cc:504] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } } peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } }
I20260929 02:52:22.517166 19238 leader_election.cc:288] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [CANDIDATE]: Term 1 election: Requested vote from peers 3e5bc99fdc974c31b6ab0b6c28035018 (127.4.197.129:43297), 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919)
I20260929 02:52:22.517562 19168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a091509dfd4c3d8216f0e86ba3059a" candidate_uuid: "e8112e9d22c74879b18abe13e333f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733e27ba999245dba0b982eeb1a361a7"
I20260929 02:52:22.517575 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a091509dfd4c3d8216f0e86ba3059a" candidate_uuid: "e8112e9d22c74879b18abe13e333f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e5bc99fdc974c31b6ab0b6c28035018"
I20260929 02:52:22.517733 19168 raft_consensus.cc:3037] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:22.517771 18901 raft_consensus.cc:3037] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:22.519850 19168 raft_consensus.cc:2454] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8112e9d22c74879b18abe13e333f5f6 in term 1.
I20260929 02:52:22.520181 18968 leader_election.cc:302] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 733e27ba999245dba0b982eeb1a361a7, e8112e9d22c74879b18abe13e333f5f6; no voters:
I20260929 02:52:22.520437 19238 raft_consensus.cc:2781] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:52:22.520778 18901 raft_consensus.cc:2454] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8112e9d22c74879b18abe13e333f5f6 in term 1.
I20260929 02:52:22.521178 19238 raft_consensus.cc:686] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 1 LEADER]: Becoming Leader. State: Replica: e8112e9d22c74879b18abe13e333f5f6, State: Running, Role: LEADER
I20260929 02:52:22.521858 19238 consensus_queue.cc:237] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } } peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } }
I20260929 02:52:22.526502 18755 catalog_manager.cc:5495] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130). New cstate: current_term: 1 leader_uuid: "e8112e9d22c74879b18abe13e333f5f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } health_report { overall_health: UNKNOWN } } }
I20260929 02:52:22.550211 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.545230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.4.197.130:46797)
0929 02:52:22.545421 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.545425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.545441 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:52:22.545522 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.545525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.545607 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.545721 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:52:22.546028 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.546738 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.548103 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.548519 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.548619 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.549071 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:52:22.549621 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:52:22.549732 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.549856 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.549927 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:52:22.558702 18901 raft_consensus.cc:1261] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Refusing update from remote peer e8112e9d22c74879b18abe13e333f5f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:52:22.558702 19168 raft_consensus.cc:1261] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7 [term 1 FOLLOWER]: Refusing update from remote peer e8112e9d22c74879b18abe13e333f5f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:52:22.559868 19242 consensus_queue.cc:1035] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:52:22.560252 19238 consensus_queue.cc:1035] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:52:22.574262 19246 mvcc.cc:205] Tried to move back new op lower bound from 7334503803109814272 to 7334503802973110272. Current Snapshot: MvccSnapshot[applied={T|T < 7334503803109814272}]
I20260929 02:52:22.586066 18757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:52:22.586313 18757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:52:22.589934 18757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Sending DeleteTablet for 3 replicas of tablet 38a091509dfd4c3d8216f0e86ba3059a
I20260929 02:52:22.591107 19148 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a091509dfd4c3d8216f0e86ba3059a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:22 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:39154
I20260929 02:52:22.591270 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 18818
I20260929 02:52:22.591202 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a091509dfd4c3d8216f0e86ba3059a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:22 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:50814
I20260929 02:52:22.592448 19015 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a091509dfd4c3d8216f0e86ba3059a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:22 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:57258
I20260929 02:52:22.616652 19255 tablet_replica.cc:330] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: stopping tablet replica
I20260929 02:52:22.616796 19257 tablet_replica.cc:330] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: stopping tablet replica
I20260929 02:52:22.617156 19257 raft_consensus.cc:2229] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:52:22.617151 19255 raft_consensus.cc:2229] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:52:22.617688 19255 raft_consensus.cc:2258] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:22.617918 19257 raft_consensus.cc:2258] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:22.620060 19255 ts_tablet_manager.cc:1905] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:52:22.627451 19254 tablet_replica.cc:330] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: stopping tablet replica
I20260929 02:52:22.627821 19255 ts_tablet_manager.cc:1918] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:52:22.627950 19254 raft_consensus.cc:2229] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:52:22.628018 19255 log.cc:1196] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/38a091509dfd4c3d8216f0e86ba3059a
I20260929 02:52:22.628500 19254 raft_consensus.cc:2258] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:22.628523 19255 ts_tablet_manager.cc:1939] T 38a091509dfd4c3d8216f0e86ba3059a P 3e5bc99fdc974c31b6ab0b6c28035018: Deleting consensus metadata
I20260929 02:52:22.630618 18744 catalog_manager.cc:4841] TS 3e5bc99fdc974c31b6ab0b6c28035018 (127.4.197.129:43297): tablet 38a091509dfd4c3d8216f0e86ba3059a (table test-table [id=8ad0a83b7a6a4a329068c8b138a898d9]) successfully deleted
I20260929 02:52:22.630710 19254 ts_tablet_manager.cc:1905] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:52:22 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:52:22.638298 19257 ts_tablet_manager.cc:1905] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:52:22.639812 19254 ts_tablet_manager.cc:1918] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:52:22.639959 19254 log.cc:1196] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/38a091509dfd4c3d8216f0e86ba3059a
I20260929 02:52:22.640414 19254 ts_tablet_manager.cc:1939] T 38a091509dfd4c3d8216f0e86ba3059a P 733e27ba999245dba0b982eeb1a361a7: Deleting consensus metadata
I20260929 02:52:22.642158 18741 catalog_manager.cc:4841] TS 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919): tablet 38a091509dfd4c3d8216f0e86ba3059a (table test-table [id=8ad0a83b7a6a4a329068c8b138a898d9]) successfully deleted
I20260929 02:52:22.645061 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.593148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.4.197.129:43297)
0929 02:52:22.593329 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.593336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.616321 (+ 22985us) server_negotiation.cc:407] Connection header received
0929 02:52:22.618442 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.618448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.618530 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.618637 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:22.620584 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.621722 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.623789 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.624427 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.624524 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.635483 (+ 10959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:22.635498 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:22.635515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:22.635565 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:22.641161 (+ 5596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:22.642614 (+ 1453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:22.642623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:22.642627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:22.642773 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:22.643245 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:22.643250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:22.643254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:22.643692 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:22.643947 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.644493 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.644704 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:52:22.646292 19257 ts_tablet_manager.cc:1918] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:52:22.646489 19257 log.cc:1196] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/38a091509dfd4c3d8216f0e86ba3059a
I20260929 02:52:22.646517 18941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:22.647122 19257 ts_tablet_manager.cc:1939] T 38a091509dfd4c3d8216f0e86ba3059a P e8112e9d22c74879b18abe13e333f5f6: Deleting consensus metadata
I20260929 02:52:22.648904 18744 catalog_manager.cc:4841] TS e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797): tablet 38a091509dfd4c3d8216f0e86ba3059a (table test-table [id=8ad0a83b7a6a4a329068c8b138a898d9]) successfully deleted
I20260929 02:52:22.734062 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 18818
I20260929 02:52:22.755285 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 18952
Sep 29 02:52:22 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:52:22.772413 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.756037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.4.197.130:46797)
0929 02:52:22.756236 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.756242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.756265 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:52:22.756411 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.756420 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.756506 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.756617 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:22.757788 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.759055 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.760312 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.760994 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.761120 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.765443 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:22.765461 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:22.765481 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:22.765535 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:22.769344 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:22.770044 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:22.770054 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:22.770058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:22.770235 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:22.770564 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:22.770569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:22.770573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:22.771253 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:22.771594 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.771755 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.771993 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:52:22.773167 19075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:22.885679 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 18952
I20260929 02:52:22.902379 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 19085
Sep 29 02:52:22 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:52:22.916352 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:22.903173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.4.197.131:33919)
0929 02:52:22.903350 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:22.903356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:22.903379 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:52:22.903582 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:22.903586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:22.903660 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:22.903756 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:22.904955 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.905654 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.907041 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:22.907420 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:22.907529 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:22.911500 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:22.911513 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:22.911527 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:22.911558 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:22.913833 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:22.914445 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:22.914452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:22.914454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:22.914581 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:22.914913 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:22.914955 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:22.914958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:22.915322 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:22.915588 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:22.915855 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 02:52:22.916052 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:52:22.916931 19208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:23.003561 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 19085
I20260929 02:52:23.016505 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 18725
I20260929 02:52:23.017542 18787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:23.129314 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 18725
I20260929 02:52:23.149394 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36783
--webserver_interface=127.4.197.190
--webserver_port=41049
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:36783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:23.472568 19262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:23.472883 19262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:23.472985 19262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:23.480356 19262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:52:23.480477 19262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:23.480533 19262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:23.480589 19262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:52:23.480634 19262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:52:23.490614 19262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:36783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=41049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:23.492065 19262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:23.493337 19262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:23.513727 19268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:23.513833 19269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:23.514384 19271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:23.514413 19262 server_base.cc:1034] running on GCE node
I20260929 02:52:23.515276 19262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:23.516697 19262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:23.517887 19262 hybrid_clock.cc:648] HybridClock initialized: now 1790650343517828 us; error 46 us; skew 500 ppm
I20260929 02:52:23.518229 19262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:23 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650343, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:23.521432 19262 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:52:23.522946 19262 webserver.cc:460] Webserver started at http://127.4.197.190:41049/ using document root <none> and password file <none>
I20260929 02:52:23.523663 19262 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:23.523777 19262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:23.529524 19262 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 02:52:23.532374 19278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:23.533689 19262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:52:23.533856 19262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "d73bb450cfbf46d0a18ac7638323a0d2"
format_stamp: "Formatted at 2026-09-29 02:52:20 on dist-test-slave-t6c1"
server_key: "f82ef543c20c094230cf0f40f353127f"
server_key_iv: "5ccf4a4dc22ecc6a39895dec67a0d8b1"
server_key_version: "encryptionkey@0"
I20260929 02:52:23.534910 19262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:23.554200 19262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:23.558486 19262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:23.558760 19262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:23.576538 19262 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:36783
I20260929 02:52:23.576557 19329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:36783 every 8 connection(s)
I20260929 02:52:23.578073 19262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:52:23.580412 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 19262
I20260929 02:52:23.581166 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:43297
--local_ip_for_outbound_sockets=127.4.197.129
--tserver_master_addrs=127.4.197.190:36783
--webserver_port=40871
--webserver_interface=127.4.197.129
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:52:23.585806 19330 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:52:23.589524 19330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Bootstrap starting.
I20260929 02:52:23.610612 19330 log.cc:824] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:23.622974 19330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:52:23.623519 19330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Bootstrap complete.
I20260929 02:52:23.632910 19330 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:23.633816 19330 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73bb450cfbf46d0a18ac7638323a0d2, State: Initialized, Role: FOLLOWER
I20260929 02:52:23.634500 19330 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:23.634753 19330 raft_consensus.cc:388] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:52:23.634888 19330 raft_consensus.cc:482] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:52:23.635113 19330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:52:23.639622 19330 raft_consensus.cc:504] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:23.640179 19330 leader_election.cc:302] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73bb450cfbf46d0a18ac7638323a0d2; no voters:
I20260929 02:52:23.640815 19330 leader_election.cc:288] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:52:23.640976 19334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:52:23.643193 19334 raft_consensus.cc:686] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 2 LEADER]: Becoming Leader. State: Replica: d73bb450cfbf46d0a18ac7638323a0d2, State: Running, Role: LEADER
I20260929 02:52:23.643262 19330 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:52:23.643791 19334 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:23.646162 19335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } } }
I20260929 02:52:23.646570 19335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:23.647265 19336 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73bb450cfbf46d0a18ac7638323a0d2. Latest consensus state: current_term: 2 leader_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } } }
I20260929 02:52:23.647663 19336 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:23.649863 19341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:52:23.658524 19341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:52:23.660173 19341 catalog_manager.cc:1259] Loaded cluster ID: f9ba519101284f7087a7c1b82d492ba1
I20260929 02:52:23.660363 19341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:52:23.663710 19341 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 02:52:23.680866 19341 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 02:52:23.690610 19341 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:52:23.692456 19341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Loaded TSK: 0
I20260929 02:52:23.694031 19341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:52:23.930172 19332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:23.930471 19332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:23.930588 19332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:23.938248 19332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:23.938370 19332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:23.938519 19332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:52:23.948733 19332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:43297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=40871
--tserver_master_addrs=127.4.197.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:23.950131 19332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:23.951532 19332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:23.966223 19358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:23.966207 19359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:23.967146 19361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:23.967226 19332 server_base.cc:1034] running on GCE node
I20260929 02:52:23.967916 19332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:23.968961 19332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:23.970151 19332 hybrid_clock.cc:648] HybridClock initialized: now 1790650343970076 us; error 69 us; skew 500 ppm
I20260929 02:52:23.970458 19332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:23 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650343, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:23.973562 19332 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:52:23.975085 19332 webserver.cc:460] Webserver started at http://127.4.197.129:40871/ using document root <none> and password file <none>
I20260929 02:52:23.975776 19332 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:23.975886 19332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:23.981667 19332 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 02:52:23.984696 19368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:23.986279 19332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:52:23.986447 19332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "3e5bc99fdc974c31b6ab0b6c28035018"
format_stamp: "Formatted at 2026-09-29 02:52:21 on dist-test-slave-t6c1"
server_key: "8aa0f29d70e1bbc532df8a90db8b9166"
server_key_iv: "aee05fb425ba48be10d841abc519fdeb"
server_key_version: "encryptionkey@0"
I20260929 02:52:23.987514 19332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:24.016809 19332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:24.020855 19332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:24.021100 19332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:24.022058 19332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:24.023557 19332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:24.023720 19332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 02:52:24.023857 19332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:24.023965 19332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:24.055241 19332 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:43297
I20260929 02:52:24.055310 19480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:43297 every 8 connection(s)
I20260929 02:52:24.056713 19332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:52:24.061913 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 19332
I20260929 02:52:24.062513 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:46797
--local_ip_for_outbound_sockets=127.4.197.130
--tserver_master_addrs=127.4.197.190:36783
--webserver_port=37633
--webserver_interface=127.4.197.130
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:52:24 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650343, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:24.081494 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:24.059987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:50039)
0929 02:52:24.060816 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:24.060863 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:24.062781 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:24.064536 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:24.064544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:24.065061 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:24.065636 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:24.065655 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.067863 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.067867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:24.069177 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:24.069190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.070272 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.070279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:24.070564 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:24.071313 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:24.071341 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:24.073846 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:24.077256 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:24.077266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:24.077281 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:24.077716 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:24.078182 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:24.078187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:24.078191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:24.078422 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:24.079474 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:24.079482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:24.079813 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 02:52:24.080049 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 02:52:24.080480 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":310,"threads_started":1}
I20260929 02:52:24.081990 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:24.059679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:50039 (local address 127.4.197.190:36783)
0929 02:52:24.060257 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:24.060270 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:24.060994 (+ 724us) server_negotiation.cc:407] Connection header received
0929 02:52:24.063469 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:24.063488 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:24.063996 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:24.064420 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:24.065876 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.067700 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.069311 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.070035 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.070180 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:24.074139 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:24.074148 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:24.074165 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:24.074204 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:24.077088 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:24.077857 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:24.077863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:24.077869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:24.078065 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:24.078566 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:24.078571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:24.078595 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:24.079315 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:24.079537 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:24.080304 (+ 767us) server_negotiation.cc:299] Negotiation successful
0929 02:52:24.080773 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":231,"threads_started":1}
I20260929 02:52:24.084870 19481 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36783
I20260929 02:52:24.085167 19481 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:24.085893 19481 heartbeater.cc:511] Master 127.4.197.190:36783 requested a full tablet report, sending...
I20260929 02:52:24.088047 19295 ts_manager.cc:194] Registered new tserver with Master: 3e5bc99fdc974c31b6ab0b6c28035018 (127.4.197.129:43297)
I20260929 02:52:24.089478 19295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:50039
W20260929 02:52:24.381117 19485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:24.381414 19485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:24.381546 19485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:24.388782 19485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:24.388922 19485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:24.389077 19485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:52:24.399030 19485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:46797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=37633
--tserver_master_addrs=127.4.197.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:24.400383 19485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:24.401836 19485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:24.415163 19493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:24.415272 19491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:24.415191 19490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:24.416019 19485 server_base.cc:1034] running on GCE node
I20260929 02:52:24.416610 19485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:24.417544 19485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:24.418751 19485 hybrid_clock.cc:648] HybridClock initialized: now 1790650344418690 us; error 44 us; skew 500 ppm
I20260929 02:52:24.419090 19485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:24 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650344, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:24.422078 19485 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:52:24.423511 19485 webserver.cc:460] Webserver started at http://127.4.197.130:37633/ using document root <none> and password file <none>
I20260929 02:52:24.424175 19485 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:24.424283 19485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:24.429338 19485 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:52:24.431867 19500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:24.433215 19485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:52:24.433373 19485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "e8112e9d22c74879b18abe13e333f5f6"
format_stamp: "Formatted at 2026-09-29 02:52:21 on dist-test-slave-t6c1"
server_key: "dfebef9a0d867bb83f2a78b2c0c02f04"
server_key_iv: "7402b3912d0122611af8c369865b7834"
server_key_version: "encryptionkey@0"
I20260929 02:52:24.434370 19485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:24.451915 19485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:24.456179 19485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:24.456445 19485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:24.457456 19485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:24.458892 19485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:24.459017 19485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 02:52:24.459126 19485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:24.459199 19485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:24.489202 19485 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:46797
I20260929 02:52:24.489257 19612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:46797 every 8 connection(s)
I20260929 02:52:24.490734 19485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:52:24.495956 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 19485
I20260929 02:52:24.496513 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:33919
--local_ip_for_outbound_sockets=127.4.197.131
--tserver_master_addrs=127.4.197.190:36783
--webserver_port=39745
--webserver_interface=127.4.197.131
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:52:24 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650344, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:24.511276 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:24.493437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:56023 (local address 127.4.197.190:36783)
0929 02:52:24.493604 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:24.493609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:24.494891 (+ 1282us) server_negotiation.cc:407] Connection header received
0929 02:52:24.496979 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:24.496986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:24.497071 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:24.497168 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:24.498649 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.499608 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.501101 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.501654 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.501782 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:24.505209 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:24.505215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:24.505219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:24.505257 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:24.508116 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:24.509007 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:24.509011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:24.509015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:24.509157 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:24.509526 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:24.509530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:24.509533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:24.509879 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:24.510036 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:24.510701 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 02:52:24.510958 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:52:24.512115 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:24.493961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:56023)
0929 02:52:24.494709 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:24.494761 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:24.496660 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:24.497415 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:24.497423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:24.497914 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:24.498491 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:24.498509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.499759 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.499762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:24.500974 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:24.500985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.501893 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.501899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:24.502169 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:24.502891 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:24.502935 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:24.505005 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:24.508307 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:24.508317 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:24.508334 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:24.508896 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:24.509254 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:24.509257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:24.509259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:24.509436 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:24.509999 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:24.510006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:24.510369 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 02:52:24.510587 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 02:52:24.511008 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":291,"threads_started":1}
I20260929 02:52:24.513600 19613 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36783
I20260929 02:52:24.513923 19613 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:24.514797 19613 heartbeater.cc:511] Master 127.4.197.190:36783 requested a full tablet report, sending...
I20260929 02:52:24.516366 19295 ts_manager.cc:194] Registered new tserver with Master: e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797)
I20260929 02:52:24.517158 19295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:56023
W20260929 02:52:24.819643 19616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:24.819904 19616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:24.820024 19616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:24.827003 19616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:24.827099 19616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:24.827251 19616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:52:24.836695 19616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:33919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=39745
--tserver_master_addrs=127.4.197.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:24.837975 19616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:24.839315 19616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:24.851416 19621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:24.851505 19624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:24.851439 19622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:24.852448 19616 server_base.cc:1034] running on GCE node
I20260929 02:52:24.853011 19616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:24.853878 19616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:24.855072 19616 hybrid_clock.cc:648] HybridClock initialized: now 1790650344855024 us; error 38 us; skew 500 ppm
I20260929 02:52:24.855348 19616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:24 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650344, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:24.858173 19616 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:52:24.859479 19616 webserver.cc:460] Webserver started at http://127.4.197.131:39745/ using document root <none> and password file <none>
I20260929 02:52:24.860105 19616 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:24.860234 19616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:24.865094 19616 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:52:24.867751 19631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:24.869087 19616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:52:24.869233 19616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "733e27ba999245dba0b982eeb1a361a7"
format_stamp: "Formatted at 2026-09-29 02:52:22 on dist-test-slave-t6c1"
server_key: "8ef1fdfa2364c2b1083934ea72d69676"
server_key_iv: "dbf890c84228ba136d07918be70f0929"
server_key_version: "encryptionkey@0"
I20260929 02:52:24.870143 19616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:24.905612 19616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:24.909435 19616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:24.909677 19616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:24.910617 19616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:24.911919 19616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:24.912011 19616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:24.912084 19616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:24.912135 19616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:24.941882 19616 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:33919
I20260929 02:52:24.941959 19743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:33919 every 8 connection(s)
I20260929 02:52:24.943315 19616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:52:24.951588 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 19616
Sep 29 02:52:24 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650344, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:24.962189 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:24.945743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:37343 (local address 127.4.197.190:36783)
0929 02:52:24.945956 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:24.945962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:24.947098 (+ 1136us) server_negotiation.cc:407] Connection header received
0929 02:52:24.948805 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:24.948810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:24.948873 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:24.948956 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:24.950278 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.951020 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.953139 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.953575 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.953692 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:24.956863 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:24.956868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:24.956871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:24.956901 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:24.959381 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:24.960018 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:24.960022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:24.960024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:24.960195 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:24.960571 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:24.960574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:24.960576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:24.960816 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:24.960970 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:24.961583 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 02:52:24.961812 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:52:24.963095 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:24.946261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:37343)
0929 02:52:24.946952 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:24.946996 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:24.948565 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:24.949139 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:24.949146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:24.949607 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:24.950156 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:24.950173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.951176 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.951181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:24.953004 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:24.953019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.953786 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.953794 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:24.954019 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:24.954636 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:24.954657 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:24.956668 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:24.959497 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:24.959505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:24.959521 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:24.959909 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:24.960295 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:24.960300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:24.960304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:24.960469 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:24.960930 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:24.960938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:24.961245 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 02:52:24.961467 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 02:52:24.961971 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":243,"threads_started":1}
I20260929 02:52:24.964380 19744 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36783
I20260929 02:52:24.964650 19744 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:24.965183 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:24.952493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.4.197.190:36783)
0929 02:52:24.953211 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:24.953217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:24.953232 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:52:24.953268 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:24.953272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:24.953340 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:24.953426 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:24.954874 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.955594 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.956943 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.957436 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.957543 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:24.959954 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:24.959959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:24.959962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:24.959989 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:24.962638 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:24.963358 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:24.963362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:24.963364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:24.963468 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:24.963835 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:24.963838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:24.963840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:24.964139 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:24.964324 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:24.964649 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 02:52:24.964861 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":474,"threads_started":1}
I20260929 02:52:24.965428 19744 heartbeater.cc:511] Master 127.4.197.190:36783 requested a full tablet report, sending...
I20260929 02:52:24.966830 19294 ts_manager.cc:194] Registered new tserver with Master: 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919)
I20260929 02:52:24.967762 19294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:37343
I20260929 02:52:24.979451 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:52:24.985599 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:24.980262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.4.197.190:36783)
0929 02:52:24.980429 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:24.980433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:24.980820 (+ 387us) server_negotiation.cc:407] Connection header received
0929 02:52:24.981003 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:24.981006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:24.981147 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:24.981252 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:52:24.981580 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.982272 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.983650 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:24.984147 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:24.984255 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:24.984636 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:52:24.985153 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:52:24.985272 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:24.985312 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 02:52:24.985376 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:52:24.986438 19294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:52:24.993816 19294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:52:25.015100 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.004645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:33919 (local address 127.0.0.1:39170)
0929 02:52:25.005536 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.005570 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:25.005731 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:25.007075 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:25.007081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:25.007124 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:25.007540 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.007553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.009628 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.009634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.012152 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.012167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.014068 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.014074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.014279 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.014333 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 02:52:25.014434 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 02:52:25.014617 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"spinlock_wait_cycles":15744,"thread_start_us":533,"threads_started":1}
I20260929 02:52:25.015883 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.004657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.0.0.1:50836)
0929 02:52:25.007406 (+ 2749us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.007422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:25.007538 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:25.008412 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:25.008417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:25.008438 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:25.008778 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.008791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.010760 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.010766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.012942 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.012957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.015162 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.015167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.015337 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.015369 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:52:25.015448 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:52:25.015561 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2607,"thread_start_us":2454,"threads_started":1}
I20260929 02:52:25.016841 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.005059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.4.197.129:43297)
0929 02:52:25.006158 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.006166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.007666 (+ 1500us) server_negotiation.cc:407] Connection header received
0929 02:52:25.007764 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.007771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.008006 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.008180 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:25.008967 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.010587 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.013131 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.014641 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.014773 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.014983 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.016296 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.016499 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":776,"threads_started":1}
I20260929 02:52:25.016908 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.004670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.4.197.131:33919)
0929 02:52:25.006580 (+ 1910us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.006586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.006608 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:52:25.006688 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.006692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.006867 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.007053 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:25.007768 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.009467 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.012372 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.013817 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.016105 (+ 2288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.016358 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.016424 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.016581 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1788,"thread_start_us":1689,"threads_started":1}
I20260929 02:52:25.020745 19679 tablet_service.cc:1467] Processing CreateTablet for tablet b4a8ffd0ceba430ab74d818f7e968ec7 (DEFAULT_TABLE table=test-table [id=2801ba845e284d20a9c9342013b81d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:25.020745 19416 tablet_service.cc:1467] Processing CreateTablet for tablet b4a8ffd0ceba430ab74d818f7e968ec7 (DEFAULT_TABLE table=test-table [id=2801ba845e284d20a9c9342013b81d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:25.022136 19416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a8ffd0ceba430ab74d818f7e968ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:25.022130 19679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a8ffd0ceba430ab74d818f7e968ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:25.023579 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.007482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:46797 (local address 127.0.0.1:57284)
0929 02:52:25.015834 (+ 8352us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.015851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:25.015947 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:25.018527 (+ 2580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:25.018532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:25.018557 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:25.018696 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.018706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.020073 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.020078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.021640 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.021652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.022877 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.022883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.023095 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.023121 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 02:52:25.023197 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 02:52:25.023324 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8256,"thread_start_us":10002,"threads_started":1}
I20260929 02:52:25.023667 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.007857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.4.197.130:46797)
0929 02:52:25.017969 (+ 10112us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.017978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.018008 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:52:25.018100 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.018107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.018330 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.018549 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:25.018829 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.019791 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.021774 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.022710 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.022831 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.023052 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.023252 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.023407 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9968,"thread_start_us":9867,"threads_started":1}
I20260929 02:52:25.026805 19548 tablet_service.cc:1467] Processing CreateTablet for tablet b4a8ffd0ceba430ab74d818f7e968ec7 (DEFAULT_TABLE table=test-table [id=2801ba845e284d20a9c9342013b81d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:25.027878 19548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a8ffd0ceba430ab74d818f7e968ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:25.037439 19760 tablet_bootstrap.cc:492] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Bootstrap starting.
I20260929 02:52:25.037992 19761 tablet_bootstrap.cc:492] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: Bootstrap starting.
I20260929 02:52:25.039706 19762 tablet_bootstrap.cc:492] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: Bootstrap starting.
I20260929 02:52:25.041136 19761 tablet_bootstrap.cc:654] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:25.041605 19760 tablet_bootstrap.cc:654] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:25.042702 19761 log.cc:824] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:25.042824 19760 log.cc:824] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:25.043069 19762 tablet_bootstrap.cc:654] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:25.044023 19762 log.cc:824] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:25.045811 19760 tablet_bootstrap.cc:492] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: No bootstrap required, opened a new log
I20260929 02:52:25.046111 19760 ts_tablet_manager.cc:1397] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:52:25.046386 19761 tablet_bootstrap.cc:492] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: No bootstrap required, opened a new log
I20260929 02:52:25.046697 19761 ts_tablet_manager.cc:1397] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:52:25.046849 19762 tablet_bootstrap.cc:492] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: No bootstrap required, opened a new log
I20260929 02:52:25.047123 19762 ts_tablet_manager.cc:1397] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 02:52:25.053177 19760 raft_consensus.cc:348] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.053506 19760 raft_consensus.cc:374] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:25.053613 19760 raft_consensus.cc:729] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5bc99fdc974c31b6ab0b6c28035018, State: Initialized, Role: FOLLOWER
I20260929 02:52:25.054148 19760 consensus_queue.cc:260] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.056797 19761 raft_consensus.cc:348] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.056972 19762 raft_consensus.cc:348] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.057257 19761 raft_consensus.cc:374] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:25.057338 19762 raft_consensus.cc:374] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:25.057435 19761 raft_consensus.cc:729] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733e27ba999245dba0b982eeb1a361a7, State: Initialized, Role: FOLLOWER
I20260929 02:52:25.057452 19762 raft_consensus.cc:729] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8112e9d22c74879b18abe13e333f5f6, State: Initialized, Role: FOLLOWER
I20260929 02:52:25.058053 19762 consensus_queue.cc:260] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.058053 19761 consensus_queue.cc:260] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.061894 19760 ts_tablet_manager.cc:1428] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260929 02:52:25.062242 19481 heartbeater.cc:503] Master 127.4.197.190:36783 was elected leader, sending a full tablet report...
I20260929 02:52:25.065704 19768 raft_consensus.cc:482] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:25.067559 19744 heartbeater.cc:503] Master 127.4.197.190:36783 was elected leader, sending a full tablet report...
I20260929 02:52:25.068773 19767 raft_consensus.cc:482] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:25.066419 19613 heartbeater.cc:503] Master 127.4.197.190:36783 was elected leader, sending a full tablet report...
I20260929 02:52:25.069398 19761 ts_tablet_manager.cc:1428] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: Time spent starting tablet: real 0.023s user 0.011s sys 0.003s
I20260929 02:52:25.069479 19768 raft_consensus.cc:504] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.069952 19767 raft_consensus.cc:504] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.072438 19767 leader_election.cc:288] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e5bc99fdc974c31b6ab0b6c28035018 (127.4.197.129:43297), e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797)
I20260929 02:52:25.073001 19768 leader_election.cc:288] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919), 3e5bc99fdc974c31b6ab0b6c28035018 (127.4.197.129:43297)
I20260929 02:52:25.070187 19762 ts_tablet_manager.cc:1428] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: Time spent starting tablet: real 0.023s user 0.016s sys 0.000s
I20260929 02:52:25.082535 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.073628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:58511 (local address 127.4.197.131:33919)
0929 02:52:25.073813 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.073820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.075208 (+ 1388us) server_negotiation.cc:407] Connection header received
0929 02:52:25.075491 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.075496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.075566 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.075626 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:25.076358 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.077290 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.079001 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.080339 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.080490 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.080560 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.082163 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.082284 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 02:52:25.085112 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.073194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:33919 (local address 127.4.197.130:58511)
0929 02:52:25.075109 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.075129 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:25.075312 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:25.075831 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:25.075836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:25.075862 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:25.076200 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.076214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.077423 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.077427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.078813 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.078824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.081788 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.081795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.081985 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.082018 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:52:25.084652 (+ 2634us) client_negotiation.cc:240] Negotiation successful
0929 02:52:25.084789 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1792,"thread_start_us":1605,"threads_started":1}
I20260929 02:52:25.085995 19699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a8ffd0ceba430ab74d818f7e968ec7" candidate_uuid: "e8112e9d22c74879b18abe13e333f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733e27ba999245dba0b982eeb1a361a7" is_pre_election: true
I20260929 02:52:25.086426 19699 raft_consensus.cc:2454] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8112e9d22c74879b18abe13e333f5f6 in term 0.
I20260929 02:52:25.087077 19501 leader_election.cc:302] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 733e27ba999245dba0b982eeb1a361a7, e8112e9d22c74879b18abe13e333f5f6; no voters:
I20260929 02:52:25.087558 19770 raft_consensus.cc:2781] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:52:25.087731 19770 raft_consensus.cc:482] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:52:25.087880 19770 raft_consensus.cc:3037] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:25.087950 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.079064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.4.197.131:51045)
0929 02:52:25.079240 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.079259 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:25.079357 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:25.081146 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:25.081151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:25.081178 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:25.081406 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.081416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.083230 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.083235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.085005 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.085020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.087014 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.087020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.087214 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.087248 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:52:25.087311 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 02:52:25.087682 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":204}
I20260929 02:52:25.088826 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.076785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:51045 (local address 127.4.197.129:43297)
0929 02:52:25.080870 (+ 4085us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.080876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.080897 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:52:25.080954 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.080959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.081023 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.081120 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:25.081710 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.083031 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.085586 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.086803 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.087498 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.088313 (+ 815us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.088360 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.088491 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":711,"server-negotiator.queue_time_us":3960,"thread_start_us":3787,"threads_started":1}
I20260929 02:52:25.089497 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.076050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:41897 (local address 127.4.197.129:43297)
0929 02:52:25.076312 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.076318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.082872 (+ 6554us) server_negotiation.cc:407] Connection header received
0929 02:52:25.083168 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.083173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.083234 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.083287 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:25.084028 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.085028 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.087302 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.088462 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.088573 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.088638 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.089153 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.089265 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260929 02:52:25.089564 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.076231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.4.197.130:41897)
0929 02:52:25.082758 (+ 6527us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.082774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:25.083011 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:25.083440 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:25.083444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:25.083466 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:25.083841 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.083852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.085196 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.085201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.087134 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.087146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.088827 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.088832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.088998 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.089032 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:52:25.089121 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 02:52:25.089242 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6388,"thread_start_us":6233,"threads_started":1}
I20260929 02:52:25.089679 19436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a8ffd0ceba430ab74d818f7e968ec7" candidate_uuid: "733e27ba999245dba0b982eeb1a361a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" is_pre_election: true
I20260929 02:52:25.090255 19436 raft_consensus.cc:2454] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733e27ba999245dba0b982eeb1a361a7 in term 0.
I20260929 02:52:25.090358 19435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a8ffd0ceba430ab74d818f7e968ec7" candidate_uuid: "e8112e9d22c74879b18abe13e333f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" is_pre_election: true
I20260929 02:52:25.090616 19435 raft_consensus.cc:2454] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8112e9d22c74879b18abe13e333f5f6 in term 0.
I20260929 02:52:25.090955 19635 leader_election.cc:302] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e5bc99fdc974c31b6ab0b6c28035018, 733e27ba999245dba0b982eeb1a361a7; no voters:
I20260929 02:52:25.091420 19769 raft_consensus.cc:2781] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:52:25.091580 19769 raft_consensus.cc:482] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:52:25.091710 19769 raft_consensus.cc:3037] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:25.091833 19770 raft_consensus.cc:504] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.092411 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.082687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:56187 (local address 127.4.197.130:46797)
0929 02:52:25.083007 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.083012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.083026 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:52:25.083080 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.083085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.083149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.083226 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:25.085528 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.086586 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.090211 (+ 3625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.091460 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.091592 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.091663 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.092142 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.092235 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260929 02:52:25.092890 19699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a8ffd0ceba430ab74d818f7e968ec7" candidate_uuid: "e8112e9d22c74879b18abe13e333f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733e27ba999245dba0b982eeb1a361a7"
I20260929 02:52:25.092975 19770 leader_election.cc:288] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [CANDIDATE]: Term 1 election: Requested vote from peers 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919), 3e5bc99fdc974c31b6ab0b6c28035018 (127.4.197.129:43297)
I20260929 02:52:25.093143 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.079375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:46797 (local address 127.4.197.131:56187)
0929 02:52:25.081021 (+ 1646us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.081037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:25.081568 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:25.085057 (+ 3489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:25.085061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:25.085087 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:25.085367 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.085378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.086851 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.086856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.090029 (+ 3173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:25.090044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.091795 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.091801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:25.091966 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.091993 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 02:52:25.092552 (+ 559us) client_negotiation.cc:240] Negotiation successful
0929 02:52:25.092696 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1520,"thread_start_us":1921,"threads_started":1}
I20260929 02:52:25.093480 19435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a8ffd0ceba430ab74d818f7e968ec7" candidate_uuid: "e8112e9d22c74879b18abe13e333f5f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e5bc99fdc974c31b6ab0b6c28035018"
I20260929 02:52:25.093652 19435 raft_consensus.cc:3037] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:25.095026 19769 raft_consensus.cc:504] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.095479 19699 raft_consensus.cc:2379] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e8112e9d22c74879b18abe13e333f5f6 in current term 1: Already voted for candidate 733e27ba999245dba0b982eeb1a361a7 in this term.
I20260929 02:52:25.095490 19568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a8ffd0ceba430ab74d818f7e968ec7" candidate_uuid: "733e27ba999245dba0b982eeb1a361a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8112e9d22c74879b18abe13e333f5f6" is_pre_election: true
I20260929 02:52:25.095687 19769 leader_election.cc:288] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [CANDIDATE]: Term 1 election: Requested vote from peers 3e5bc99fdc974c31b6ab0b6c28035018 (127.4.197.129:43297), e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797)
I20260929 02:52:25.095916 19568 raft_consensus.cc:2379] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 733e27ba999245dba0b982eeb1a361a7 in current term 1: Already voted for candidate e8112e9d22c74879b18abe13e333f5f6 in this term.
I20260929 02:52:25.096572 19568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a8ffd0ceba430ab74d818f7e968ec7" candidate_uuid: "733e27ba999245dba0b982eeb1a361a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8112e9d22c74879b18abe13e333f5f6"
I20260929 02:52:25.096639 19435 raft_consensus.cc:2454] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8112e9d22c74879b18abe13e333f5f6 in term 1.
I20260929 02:52:25.096799 19568 raft_consensus.cc:2379] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 733e27ba999245dba0b982eeb1a361a7 in current term 1: Already voted for candidate e8112e9d22c74879b18abe13e333f5f6 in this term.
I20260929 02:52:25.097095 19436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a8ffd0ceba430ab74d818f7e968ec7" candidate_uuid: "733e27ba999245dba0b982eeb1a361a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e5bc99fdc974c31b6ab0b6c28035018"
I20260929 02:52:25.097193 19504 leader_election.cc:302] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e5bc99fdc974c31b6ab0b6c28035018, e8112e9d22c74879b18abe13e333f5f6; no voters: 733e27ba999245dba0b982eeb1a361a7
I20260929 02:52:25.097335 19436 raft_consensus.cc:2379] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 733e27ba999245dba0b982eeb1a361a7 in current term 1: Already voted for candidate e8112e9d22c74879b18abe13e333f5f6 in this term.
I20260929 02:52:25.097442 19770 raft_consensus.cc:2781] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:52:25.097602 19770 raft_consensus.cc:686] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 1 LEADER]: Becoming Leader. State: Replica: e8112e9d22c74879b18abe13e333f5f6, State: Running, Role: LEADER
I20260929 02:52:25.097859 19635 leader_election.cc:302] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 733e27ba999245dba0b982eeb1a361a7; no voters: 3e5bc99fdc974c31b6ab0b6c28035018, e8112e9d22c74879b18abe13e333f5f6
I20260929 02:52:25.098062 19769 raft_consensus.cc:2726] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 02:52:25.098032 19770 consensus_queue.cc:237] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:25.101662 19294 catalog_manager.cc:5495] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130). New cstate: current_term: 1 leader_uuid: "e8112e9d22c74879b18abe13e333f5f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } health_report { overall_health: HEALTHY } } }
I20260929 02:52:25.135812 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.130951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.4.197.130:46797)
0929 02:52:25.131100 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.131104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.131142 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:52:25.131221 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.131224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.131298 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.131415 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:52:25.131676 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.132400 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.133704 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.134190 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.134304 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.134729 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:52:25.135319 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:52:25.135437 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.135485 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.135556 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:52:25.143537 19699 raft_consensus.cc:1261] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 1 FOLLOWER]: Refusing update from remote peer e8112e9d22c74879b18abe13e333f5f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:52:25.143537 19436 raft_consensus.cc:1261] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Refusing update from remote peer e8112e9d22c74879b18abe13e333f5f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:52:25.144256 19768 consensus_queue.cc:1035] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:52:25.144550 19770 consensus_queue.cc:1035] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:52:25.160351 19778 mvcc.cc:205] Tried to move back new op lower bound from 7334503813699149824 to 7334503813524979712. Current Snapshot: MvccSnapshot[applied={T|T < 7334503813699149824 or (T in {7334503813699149824})}]
I20260929 02:52:25.171064 19294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:52:25.171327 19294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:52:25.175056 19294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Sending DeleteTablet for 3 replicas of tablet b4a8ffd0ceba430ab74d818f7e968ec7
I20260929 02:52:25.176244 19679 tablet_service.cc:1514] Processing DeleteTablet for tablet b4a8ffd0ceba430ab74d818f7e968ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:25 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:39170
I20260929 02:52:25.176539 19548 tablet_service.cc:1514] Processing DeleteTablet for tablet b4a8ffd0ceba430ab74d818f7e968ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:25 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:57284
I20260929 02:52:25.176681 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 19332
I20260929 02:52:25.176999 19416 tablet_service.cc:1514] Processing DeleteTablet for tablet b4a8ffd0ceba430ab74d818f7e968ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:25 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:50836
I20260929 02:52:25.186501 19786 tablet_replica.cc:330] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: stopping tablet replica
I20260929 02:52:25.187121 19786 raft_consensus.cc:2229] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:52:25.187882 19786 raft_consensus.cc:2258] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:25.191622 19788 tablet_replica.cc:330] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: stopping tablet replica
I20260929 02:52:25.192133 19788 raft_consensus.cc:2229] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:52:25.192783 19788 raft_consensus.cc:2258] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:25.193992 19787 tablet_replica.cc:330] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: stopping tablet replica
I20260929 02:52:25.194448 19787 raft_consensus.cc:2229] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:52:25.195217 19787 raft_consensus.cc:2258] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:25.196098 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.178327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.4.197.129:43297)
0929 02:52:25.178701 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.178708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.178729 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:52:25.178792 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.178798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.178887 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.179013 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:25.181104 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.182307 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.184396 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.185060 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.185181 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.188932 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:25.188952 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:25.188971 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:25.189015 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:25.193010 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:25.193878 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:25.193887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:25.193890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:25.194118 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:25.194561 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:25.194568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:25.194571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:25.195095 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:25.195401 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.195702 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.195861 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260929 02:52:25.196808 19786 ts_tablet_manager.cc:1905] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:52:25.196977 19476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:25.197749 19787 ts_tablet_manager.cc:1905] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:52:25.203560 19786 ts_tablet_manager.cc:1918] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:52:25.203794 19786 log.cc:1196] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/b4a8ffd0ceba430ab74d818f7e968ec7
I20260929 02:52:25.204350 19786 ts_tablet_manager.cc:1939] T b4a8ffd0ceba430ab74d818f7e968ec7 P 733e27ba999245dba0b982eeb1a361a7: Deleting consensus metadata
I20260929 02:52:25.204681 19787 ts_tablet_manager.cc:1918] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:52:25.204873 19787 log.cc:1196] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/b4a8ffd0ceba430ab74d818f7e968ec7
I20260929 02:52:25.205497 19787 ts_tablet_manager.cc:1939] T b4a8ffd0ceba430ab74d818f7e968ec7 P e8112e9d22c74879b18abe13e333f5f6: Deleting consensus metadata
I20260929 02:52:25.206426 19279 catalog_manager.cc:4841] TS 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919): tablet b4a8ffd0ceba430ab74d818f7e968ec7 (table test-table [id=2801ba845e284d20a9c9342013b81d50]) successfully deleted
I20260929 02:52:25.207077 19282 catalog_manager.cc:4841] TS e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797): tablet b4a8ffd0ceba430ab74d818f7e968ec7 (table test-table [id=2801ba845e284d20a9c9342013b81d50]) successfully deleted
I20260929 02:52:25.197144 19788 ts_tablet_manager.cc:1905] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:52:25.296844 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 19332
W20260929 02:52:25.317149 19282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.197.129:43297 (error 108)
I20260929 02:52:25.318401 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 19485
I20260929 02:52:25.319021 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.318464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.0.0.1:50860)
0929 02:52:25.318726 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.318836 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260929 02:52:25.319420 19282 catalog_manager.cc:4584] TS 3e5bc99fdc974c31b6ab0b6c28035018 (127.4.197.129:43297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4a8ffd0ceba430ab74d818f7e968ec7: Network error: Client connection negotiation failed: client connection to 127.4.197.129:43297: connect: Connection refused (error 111)
I20260929 02:52:25.333738 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.319191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.4.197.130:46797)
0929 02:52:25.319377 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.319383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.319476 (+ 93us) server_negotiation.cc:407] Connection header received
0929 02:52:25.319671 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.319677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.319753 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.319850 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:25.321562 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.322364 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.324272 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.324673 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.324774 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.327945 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:25.327957 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:25.327970 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:25.328007 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:25.330951 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:25.331802 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:25.331808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:25.331810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:25.331949 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:25.332360 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:25.332368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:25.332370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:25.332774 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:25.332996 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.333280 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.333448 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:52:25.334376 19608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:25.373117 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.372630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.0.0.1:50864)
0929 02:52:25.372855 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.372957 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 02:52:25.374155 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.373770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.0.0.1:50878)
0929 02:52:25.373929 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.374017 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 02:52:25.434566 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.434082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.0.0.1:50892)
0929 02:52:25.434330 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.434423 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260929 02:52:25.435577 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.435211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.0.0.1:50902)
0929 02:52:25.435359 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.435442 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:52:25.444161 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 19485
I20260929 02:52:25.458485 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 19616
I20260929 02:52:25.475244 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.463198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.4.197.131:33919)
0929 02:52:25.463371 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:25.463376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:25.463395 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:52:25.463514 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:25.463520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:25.463597 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:25.463681 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:25.464943 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.465691 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.467125 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:25.467566 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:25.467674 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:25.469982 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:25.469995 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:25.470010 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:25.470040 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:25.472608 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:25.473247 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:25.473256 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:25.473259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:25.473405 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:25.473738 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:25.473743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:25.473746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:25.474100 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:25.474365 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:25.474684 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 02:52:25.474901 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:52:25.475847 19739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:25.516559 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.516069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.0.0.1:50916)
0929 02:52:25.516288 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.516386 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:52:25.517645 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:25.517222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43297 (local address 127.0.0.1:50922)
0929 02:52:25.517449 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:25.517525 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 02:52:25.563382 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 19616
I20260929 02:52:25.577284 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 19262
I20260929 02:52:25.578420 19325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:25.687446 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 19262
I20260929 02:52:25.704452 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36783
--webserver_interface=127.4.197.190
--webserver_port=41049
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:36783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:26.007927 19794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:26.008213 19794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:26.008313 19794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:26.015385 19794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:52:26.015491 19794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:26.015534 19794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:26.015590 19794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:52:26.015632 19794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:52:26.025189 19794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:36783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=41049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:26.026394 19794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:26.027616 19794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:26.038393 19799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:26.038434 19800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:26.038456 19802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:26.039237 19794 server_base.cc:1034] running on GCE node
I20260929 02:52:26.039907 19794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:26.041218 19794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:26.042423 19794 hybrid_clock.cc:648] HybridClock initialized: now 1790650346042369 us; error 42 us; skew 500 ppm
I20260929 02:52:26.042754 19794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:26 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650346, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:26.045598 19794 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:52:26.046976 19794 webserver.cc:460] Webserver started at http://127.4.197.190:41049/ using document root <none> and password file <none>
I20260929 02:52:26.047613 19794 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:26.047736 19794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:26.052968 19794 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:52:26.055720 19809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:26.056984 19794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:52:26.057133 19794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "d73bb450cfbf46d0a18ac7638323a0d2"
format_stamp: "Formatted at 2026-09-29 02:52:20 on dist-test-slave-t6c1"
server_key: "f82ef543c20c094230cf0f40f353127f"
server_key_iv: "5ccf4a4dc22ecc6a39895dec67a0d8b1"
server_key_version: "encryptionkey@0"
I20260929 02:52:26.058068 19794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:26.072993 19794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:26.076620 19794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:26.076860 19794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:26.093412 19794 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:36783
I20260929 02:52:26.093438 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:36783 every 8 connection(s)
I20260929 02:52:26.094780 19794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:52:26.101258 19861 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:52:26.102710 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 19794
I20260929 02:52:26.103698 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:43297
--local_ip_for_outbound_sockets=127.4.197.129
--tserver_master_addrs=127.4.197.190:36783
--webserver_port=40871
--webserver_interface=127.4.197.129
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:52:26.105350 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Bootstrap starting.
I20260929 02:52:26.127272 19861 log.cc:824] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:26.143038 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:52:26.143507 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2: Bootstrap complete.
I20260929 02:52:26.152089 19861 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:26.152923 19861 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73bb450cfbf46d0a18ac7638323a0d2, State: Initialized, Role: FOLLOWER
I20260929 02:52:26.153496 19861 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:26.153704 19861 raft_consensus.cc:388] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:52:26.153812 19861 raft_consensus.cc:482] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:52:26.153981 19861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:52:26.157630 19861 raft_consensus.cc:504] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:26.158090 19861 leader_election.cc:302] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73bb450cfbf46d0a18ac7638323a0d2; no voters:
I20260929 02:52:26.158669 19861 leader_election.cc:288] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 02:52:26.158804 19865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:52:26.159796 19865 raft_consensus.cc:686] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [term 3 LEADER]: Becoming Leader. State: Replica: d73bb450cfbf46d0a18ac7638323a0d2, State: Running, Role: LEADER
I20260929 02:52:26.160390 19865 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } }
I20260929 02:52:26.160724 19861 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:52:26.162799 19867 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73bb450cfbf46d0a18ac7638323a0d2. Latest consensus state: current_term: 3 leader_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } } }
I20260929 02:52:26.163172 19867 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:26.163523 19866 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bb450cfbf46d0a18ac7638323a0d2" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36783 } } }
I20260929 02:52:26.163769 19866 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73bb450cfbf46d0a18ac7638323a0d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:26.172380 19872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:52:26.181481 19872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:52:26.183028 19872 catalog_manager.cc:1259] Loaded cluster ID: f9ba519101284f7087a7c1b82d492ba1
I20260929 02:52:26.183175 19872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 02:52:26.185196 19872 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdb5e25f9ad google::LogMessage::Fail() at ??:0
@ 0x7fdb5e26288a google::LogMessage::SendToLog() at ??:0
@ 0x7fdb5e25f4de google::LogMessage::Flush() at ??:0
@ 0x7fdb5e260a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdb73c92512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fdb73c8f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdb73d17610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fdb73d17439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fdb60eab345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdb60eae91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fdb60eae799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fdb60e89eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdb641636db start_thread at ??:0
@ 0x7fdb5d0f961f clone at ??:0
W20260929 02:52:26.442576 19863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:26.442874 19863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:26.443030 19863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:26.450027 19863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:26.450141 19863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:26.450285 19863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:52:26.463004 19863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:43297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=40871
--tserver_master_addrs=127.4.197.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:26.464584 19863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:26.466238 19863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:26.479897 19900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:26.479820 19901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:26.479820 19903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:26.480511 19863 server_base.cc:1034] running on GCE node
I20260929 02:52:26.481091 19863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:26.481920 19863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:26.483101 19863 hybrid_clock.cc:648] HybridClock initialized: now 1790650346483062 us; error 33 us; skew 500 ppm
I20260929 02:52:26.483376 19863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:26 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650346, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:26.486219 19863 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:52:26.487710 19863 webserver.cc:460] Webserver started at http://127.4.197.129:40871/ using document root <none> and password file <none>
I20260929 02:52:26.488353 19863 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:26.488466 19863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:26.493409 19863 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:52:26.496261 19910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:26.497552 19863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:52:26.497692 19863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "3e5bc99fdc974c31b6ab0b6c28035018"
format_stamp: "Formatted at 2026-09-29 02:52:21 on dist-test-slave-t6c1"
server_key: "8aa0f29d70e1bbc532df8a90db8b9166"
server_key_iv: "aee05fb425ba48be10d841abc519fdeb"
server_key_version: "encryptionkey@0"
I20260929 02:52:26.498591 19863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:26.513290 19863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:26.517050 19863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:26.517288 19863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:26.518152 19863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:26.519820 19917 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 02:52:26.524978 19863 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 02:52:26.525085 19863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260929 02:52:26.525190 19863 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 02:52:26.528167 19863 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 02:52:26.528239 19863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.001s
I20260929 02:52:26.528544 19917 tablet_bootstrap.cc:492] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Bootstrap starting.
I20260929 02:52:26.536377 19917 log.cc:824] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:26.550263 19917 tablet_bootstrap.cc:492] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:52:26.550834 19917 tablet_bootstrap.cc:492] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Bootstrap complete.
I20260929 02:52:26.551281 19917 ts_tablet_manager.cc:1397] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20260929 02:52:26.560105 19917 raft_consensus.cc:348] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:26.561445 19917 raft_consensus.cc:729] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5bc99fdc974c31b6ab0b6c28035018, State: Initialized, Role: FOLLOWER
I20260929 02:52:26.562074 19917 consensus_queue.cc:260] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:26.563830 19917 ts_tablet_manager.cc:1428] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260929 02:52:26.566777 19863 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:43297
I20260929 02:52:26.566843 20025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:43297 every 8 connection(s)
I20260929 02:52:26.567916 19863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:52:26.572088 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:26.570328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:35815)
0929 02:52:26.571078 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:26.571290 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":281,"threads_started":1}
W20260929 02:52:26.572603 20026 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:36783 (0 consecutive failures): Network error: Failed to ping master at 127.4.197.190:36783: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
I20260929 02:52:26.573290 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:26.573026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:51869)
0929 02:52:26.573135 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:26.573195 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 02:52:26.573958 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:26.573682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:59379)
0929 02:52:26.573788 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:26.573852 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 02:52:26.574139 20026 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:52:26.576926 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 19863
I20260929 02:52:26.577451 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:46797
--local_ip_for_outbound_sockets=127.4.197.130
--tserver_master_addrs=127.4.197.190:36783
--webserver_port=37633
--webserver_interface=127.4.197.130
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:26.883034 20029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:26.883291 20029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:26.883381 20029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:26.890240 20029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:26.890333 20029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:26.890437 20029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:52:26.899597 20029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:46797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=37633
--tserver_master_addrs=127.4.197.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:26.900712 20029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:26.901947 20029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:26.914189 20034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:26.914237 20037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:26.914275 20035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:26.915132 20029 server_base.cc:1034] running on GCE node
I20260929 02:52:26.915714 20029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:26.916522 20029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:26.917716 20029 hybrid_clock.cc:648] HybridClock initialized: now 1790650346917661 us; error 42 us; skew 500 ppm
I20260929 02:52:26.917973 20029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:26 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650346, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:26.920861 20029 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:52:26.922155 20029 webserver.cc:460] Webserver started at http://127.4.197.130:37633/ using document root <none> and password file <none>
I20260929 02:52:26.922793 20029 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:26.922910 20029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:26.927845 20029 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:52:26.932446 20044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:26.933660 20029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:52:26.933800 20029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "e8112e9d22c74879b18abe13e333f5f6"
format_stamp: "Formatted at 2026-09-29 02:52:21 on dist-test-slave-t6c1"
server_key: "dfebef9a0d867bb83f2a78b2c0c02f04"
server_key_iv: "7402b3912d0122611af8c369865b7834"
server_key_version: "encryptionkey@0"
I20260929 02:52:26.934687 20029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:26.957399 20029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:26.961229 20029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:26.961460 20029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:26.962352 20029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:26.963604 20029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:26.963689 20029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:26.963788 20029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:26.963850 20029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:26.993582 20029 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:46797
I20260929 02:52:26.993649 20156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:46797 every 8 connection(s)
I20260929 02:52:26.994979 20029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:52:26.996757 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 20029
I20260929 02:52:26.997321 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:33919
--local_ip_for_outbound_sockets=127.4.197.131
--tserver_master_addrs=127.4.197.190:36783
--webserver_port=39745
--webserver_interface=127.4.197.131
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:52:27.000142 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:26.997944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:34497)
0929 02:52:26.998758 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:26.999081 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":308,"threads_started":1}
W20260929 02:52:27.000849 20157 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:36783 (0 consecutive failures): Network error: Failed to ping master at 127.4.197.190:36783: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
I20260929 02:52:27.001669 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:27.001316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:34479)
0929 02:52:27.001467 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:27.001546 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:52:27.002559 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:27.002231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:54801)
0929 02:52:27.002350 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:27.002432 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 02:52:27.002771 20157 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:52:27.301265 20160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:27.301510 20160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:27.301625 20160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:27.308570 20160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:27.308662 20160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:27.308777 20160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:52:27.317922 20160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:33919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=39745
--tserver_master_addrs=127.4.197.190:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:27.319114 20160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:27.320319 20160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:27.332114 20165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:27.332137 20166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:27.332212 20168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:27.333067 20160 server_base.cc:1034] running on GCE node
I20260929 02:52:27.333590 20160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:27.334398 20160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:27.335579 20160 hybrid_clock.cc:648] HybridClock initialized: now 1790650347335538 us; error 39 us; skew 500 ppm
I20260929 02:52:27.335862 20160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:27 dist-test-slave-t6c1 krb5kdc[18712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650347, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:27.342624 20160 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:52:27.344031 20160 webserver.cc:460] Webserver started at http://127.4.197.131:39745/ using document root <none> and password file <none>
I20260929 02:52:27.344660 20160 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:27.344770 20160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:27.349831 20160 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:52:27.352320 20175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:27.353475 20160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:52:27.353615 20160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "733e27ba999245dba0b982eeb1a361a7"
format_stamp: "Formatted at 2026-09-29 02:52:22 on dist-test-slave-t6c1"
server_key: "8ef1fdfa2364c2b1083934ea72d69676"
server_key_iv: "dbf890c84228ba136d07918be70f0929"
server_key_version: "encryptionkey@0"
I20260929 02:52:27.354488 20160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:27.368520 20160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:27.372208 20160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:27.372442 20160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:27.373340 20160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:27.374577 20160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:27.374665 20160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:27.374766 20160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:27.374846 20160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:27.403694 20160 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:33919
I20260929 02:52:27.403754 20287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:33919 every 8 connection(s)
I20260929 02:52:27.405012 20160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:52:27.407404 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 20160
I20260929 02:52:27.409858 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:27.407889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:42297)
0929 02:52:27.408711 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:27.409011 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":295,"threads_started":1}
W20260929 02:52:27.410504 20288 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:36783 (0 consecutive failures): Network error: Failed to ping master at 127.4.197.190:36783: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
I20260929 02:52:27.411213 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:27.410902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:49061)
0929 02:52:27.411049 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:27.411112 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:52:27.411944 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:27.411654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:47237)
0929 02:52:27.411778 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:27.411847 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 02:52:27.412147 20288 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:52:27.575399 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:27.574619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:47789)
0929 02:52:27.575090 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:27.575180 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":218,"threads_started":1}
I20260929 02:52:27.933456 20293 raft_consensus.cc:482] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:27.933724 20293 raft_consensus.cc:504] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:27.935566 20293 leader_election.cc:288] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919), e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797)
Sep 29 02:52:27 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650346, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
Sep 29 02:52:27 dist-test-slave-t6c1 krb5kdc[18712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650346, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:52:27.955775 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:27.935813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:46797 (local address 127.4.197.129:42411)
0929 02:52:27.936103 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:27.936149 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:27.936774 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:27.940579 (+ 3805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:27.940587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:27.941340 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:27.942033 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:27.942051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:27.943537 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:27.943541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:27.944891 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:27.944903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:27.945826 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:27.945835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:27.946063 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:27.946731 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:27.946752 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:27.948932 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:27.952280 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:27.952288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:27.952302 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:27.952808 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:27.953265 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:27.953270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:27.953273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:27.953495 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:27.954279 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:27.954286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:27.954612 (+ 326us) client_negotiation.cc:769] Sending connection context
0929 02:52:27.954849 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 02:52:27.955348 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":198,"spinlock_wait_cycles":9216}
I20260929 02:52:27.955827 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:27.936219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:42411 (local address 127.4.197.130:46797)
0929 02:52:27.938018 (+ 1799us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:27.938033 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:27.938069 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:52:27.939623 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:27.939631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:27.940103 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:27.940460 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:27.942213 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:27.943369 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:27.945019 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:27.945603 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:27.945770 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:27.949230 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:27.949240 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:27.949256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:27.949298 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:27.952144 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:27.952946 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:27.952955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:27.952962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:27.953149 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:27.953598 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:27.953602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:27.953606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:27.954133 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:27.954265 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:27.955106 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 02:52:27.955420 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":1522,"threads_started":1}
I20260929 02:52:27.955880 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:27.935850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:33919 (local address 127.4.197.129:48819)
0929 02:52:27.937874 (+ 2024us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:27.937896 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:27.938039 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:27.940577 (+ 2538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:27.940587 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:27.941340 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:27.942035 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:27.942052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:27.943531 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:27.943535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:27.944888 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:27.944899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:27.945778 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:27.945785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:27.946060 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:27.946784 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:27.946799 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:27.949361 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:27.952680 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:27.952684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:27.952686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:27.953151 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:27.953600 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:27.953605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:27.953609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:27.953770 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:27.954450 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:27.954459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:27.954609 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 02:52:27.954850 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 02:52:27.955476 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1898,"mutex_wait_us":357,"spinlock_wait_cycles":80512,"thread_start_us":1734,"threads_started":1}
I20260929 02:52:27.955798 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:27.936004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:48819 (local address 127.4.197.131:33919)
0929 02:52:27.937577 (+ 1573us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:27.937593 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:27.938016 (+ 423us) server_negotiation.cc:407] Connection header received
0929 02:52:27.939742 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:27.939750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:27.940115 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:27.940460 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:27.942210 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:27.943388 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:27.945007 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:27.945587 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:27.945733 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:27.949572 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:27.949579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:27.949589 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:27.949621 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:27.952510 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:27.953266 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:27.953274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:27.953280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:27.953480 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:27.953862 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:27.953865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:27.953867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:27.954313 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:27.954469 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:27.955100 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 02:52:27.955434 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1368,"thread_start_us":1261,"threads_started":1}
W20260929 02:52:27.958954 19914 leader_election.cc:341] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797): Not found: Tablet not found: b4a8ffd0ceba430ab74d818f7e968ec7
W20260929 02:52:27.959213 19911 leader_election.cc:341] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919): Not found: Tablet not found: b4a8ffd0ceba430ab74d818f7e968ec7
I20260929 02:52:27.959371 19914 leader_election.cc:302] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e5bc99fdc974c31b6ab0b6c28035018; no voters: 733e27ba999245dba0b982eeb1a361a7, e8112e9d22c74879b18abe13e333f5f6
I20260929 02:52:27.959754 20293 raft_consensus.cc:2726] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:52:28.004114 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:28.003306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:39889)
0929 02:52:28.003761 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:28.003902 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":205,"threads_started":1}
I20260929 02:52:28.413470 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:28.412632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:45729)
0929 02:52:28.413119 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:28.413251 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":262,"threads_started":1}
I20260929 02:52:28.576872 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:28.576230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:58029)
0929 02:52:28.576621 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:28.576705 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":187,"threads_started":1}
I20260929 02:52:29.005548 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:29.004919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:42795)
0929 02:52:29.005312 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:29.005394 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":200,"threads_started":1}
I20260929 02:52:29.414956 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:29.414229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:46979)
0929 02:52:29.414680 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:29.414781 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":245,"threads_started":1}
I20260929 02:52:29.578557 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:29.577774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:53285)
0929 02:52:29.578265 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:29.578364 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
I20260929 02:52:29.941411 20303 raft_consensus.cc:482] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:29.941597 20303 raft_consensus.cc:504] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:29.942169 20303 leader_election.cc:288] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919), e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797)
W20260929 02:52:29.943183 19914 leader_election.cc:341] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797): Not found: Tablet not found: b4a8ffd0ceba430ab74d818f7e968ec7
W20260929 02:52:29.943339 19911 leader_election.cc:341] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919): Not found: Tablet not found: b4a8ffd0ceba430ab74d818f7e968ec7
I20260929 02:52:29.943490 19911 leader_election.cc:302] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e5bc99fdc974c31b6ab0b6c28035018; no voters: 733e27ba999245dba0b982eeb1a361a7, e8112e9d22c74879b18abe13e333f5f6
I20260929 02:52:29.943730 20303 raft_consensus.cc:2726] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:52:30.007123 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:30.006401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:51229)
0929 02:52:30.006841 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:30.006954 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":217,"threads_started":1}
I20260929 02:52:30.416523 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:30.415779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:48265)
0929 02:52:30.416264 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:30.416348 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":220,"threads_started":1}
I20260929 02:52:30.580297 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:30.579447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:52797)
0929 02:52:30.579962 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:30.580077 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":262,"threads_started":1}
I20260929 02:52:31.008598 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:31.007944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:52457)
0929 02:52:31.008346 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:31.008426 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":191,"threads_started":1}
I20260929 02:52:31.418068 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:31.417399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:46355)
0929 02:52:31.417805 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:31.417895 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":209,"threads_started":1}
I20260929 02:52:31.582405 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:31.581445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:54581)
0929 02:52:31.582119 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:31.582213 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":418,"threads_started":1}
I20260929 02:52:31.925292 20310 raft_consensus.cc:482] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:31.925487 20310 raft_consensus.cc:504] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:31.926146 20310 leader_election.cc:288] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919), e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797)
W20260929 02:52:31.927076 19911 leader_election.cc:341] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919): Not found: Tablet not found: b4a8ffd0ceba430ab74d818f7e968ec7
W20260929 02:52:31.927258 19914 leader_election.cc:341] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797): Not found: Tablet not found: b4a8ffd0ceba430ab74d818f7e968ec7
I20260929 02:52:31.927403 19914 leader_election.cc:302] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e5bc99fdc974c31b6ab0b6c28035018; no voters: 733e27ba999245dba0b982eeb1a361a7, e8112e9d22c74879b18abe13e333f5f6
I20260929 02:52:31.927644 20310 raft_consensus.cc:2726] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:52:32.010071 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:32.009416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:54529)
0929 02:52:32.009832 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:32.009909 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":212,"threads_started":1}
I20260929 02:52:32.419770 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:32.419037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:48229)
0929 02:52:32.419523 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:32.419606 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":275,"threads_started":1}
I20260929 02:52:32.584020 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:32.583309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:40407)
0929 02:52:32.583763 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:32.583856 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":230,"threads_started":1}
I20260929 02:52:33.011838 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:33.011046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:47961)
0929 02:52:33.011539 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:33.011639 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":244,"threads_started":1}
I20260929 02:52:33.421315 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:33.420595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:52255)
0929 02:52:33.421058 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:33.421133 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
I20260929 02:52:33.585582 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:33.584857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:46323)
0929 02:52:33.585294 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:33.585374 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":239,"threads_started":1}
I20260929 02:52:34.013475 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:34.012701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:42547)
0929 02:52:34.013205 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:34.013288 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":230,"threads_started":1}
I20260929 02:52:34.277297 20318 raft_consensus.cc:482] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:34.277493 20318 raft_consensus.cc:504] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:34.278153 20318 leader_election.cc:288] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919), e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797)
W20260929 02:52:34.279062 19914 leader_election.cc:341] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797): Not found: Tablet not found: b4a8ffd0ceba430ab74d818f7e968ec7
W20260929 02:52:34.279253 19911 leader_election.cc:341] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919): Not found: Tablet not found: b4a8ffd0ceba430ab74d818f7e968ec7
I20260929 02:52:34.279353 19911 leader_election.cc:302] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e5bc99fdc974c31b6ab0b6c28035018; no voters: 733e27ba999245dba0b982eeb1a361a7, e8112e9d22c74879b18abe13e333f5f6
I20260929 02:52:34.279579 20318 raft_consensus.cc:2726] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:52:34.423038 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:34.422203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:59083)
0929 02:52:34.422720 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:34.422803 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":213,"threads_started":1}
I20260929 02:52:34.587522 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:34.586577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:57567)
0929 02:52:34.587198 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:34.587333 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":335,"threads_started":1}
I20260929 02:52:35.015358 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:35.014472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:39199)
0929 02:52:35.014987 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:35.015080 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
I20260929 02:52:35.424678 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:35.423931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:33807)
0929 02:52:35.424396 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:35.424479 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":255,"threads_started":1}
I20260929 02:52:35.589128 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:35.588411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:38317)
0929 02:52:35.588871 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:35.588953 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":218,"threads_started":1}
I20260929 02:52:36.016930 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:36.016219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:52851)
0929 02:52:36.016685 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:36.016778 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":224,"threads_started":1}
I20260929 02:52:36.426282 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:36.425540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:41571)
0929 02:52:36.426028 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:36.426119 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":240,"threads_started":1}
I20260929 02:52:36.591013 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:36.590218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:46243)
0929 02:52:36.590719 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:36.590812 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":277,"threads_started":1}
I20260929 02:52:37.018433 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:37.017749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:60559)
0929 02:52:37.018164 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:37.018260 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":228,"threads_started":1}
I20260929 02:52:37.347182 20328 raft_consensus.cc:482] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:37.347373 20328 raft_consensus.cc:504] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733e27ba999245dba0b982eeb1a361a7" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 33919 } } peers { permanent_uuid: "3e5bc99fdc974c31b6ab0b6c28035018" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43297 } } peers { permanent_uuid: "e8112e9d22c74879b18abe13e333f5f6" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 46797 } }
I20260929 02:52:37.347998 20328 leader_election.cc:288] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919), e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797)
W20260929 02:52:37.348918 19914 leader_election.cc:341] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8112e9d22c74879b18abe13e333f5f6 (127.4.197.130:46797): Not found: Tablet not found: b4a8ffd0ceba430ab74d818f7e968ec7
W20260929 02:52:37.349092 19911 leader_election.cc:341] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 733e27ba999245dba0b982eeb1a361a7 (127.4.197.131:33919): Not found: Tablet not found: b4a8ffd0ceba430ab74d818f7e968ec7
I20260929 02:52:37.349241 19911 leader_election.cc:302] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e5bc99fdc974c31b6ab0b6c28035018; no voters: 733e27ba999245dba0b982eeb1a361a7, e8112e9d22c74879b18abe13e333f5f6
I20260929 02:52:37.349517 20328 raft_consensus.cc:2726] T b4a8ffd0ceba430ab74d818f7e968ec7 P 3e5bc99fdc974c31b6ab0b6c28035018 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:52:37.428174 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:37.427285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:40207)
0929 02:52:37.427799 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:37.427915 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":270,"threads_started":1}
I20260929 02:52:37.592900 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:37.591973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:57671)
0929 02:52:37.592509 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:37.592618 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":267,"threads_started":1}
I20260929 02:52:38.020006 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:38.019251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:54315)
0929 02:52:38.019740 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:38.019833 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":240,"threads_started":1}
I20260929 02:52:38.429911 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:38.429122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:57973)
0929 02:52:38.429642 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:38.429741 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":241,"threads_started":1}
I20260929 02:52:38.594579 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:38.593846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:39271)
0929 02:52:38.594307 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:38.594406 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":215,"threads_started":1}
I20260929 02:52:39.021588 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:39.020887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:35111)
0929 02:52:39.021336 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:39.021420 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":222,"threads_started":1}
I20260929 02:52:39.431465 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:39.430754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:54651)
0929 02:52:39.431209 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:39.431300 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":251,"threads_started":1}
I20260929 02:52:39.596244 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:39.595509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:41271)
0929 02:52:39.595983 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:39.596072 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":252,"threads_started":1}
I20260929 02:52:40.023279 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:40.022456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:56883)
0929 02:52:40.022999 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:40.023106 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":255,"threads_started":1}
I20260929 02:52:40.433063 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:40.432339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:52173)
0929 02:52:40.432801 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:40.432893 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":231,"threads_started":1}
I20260929 02:52:40.597967 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:40.597187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:35775)
0929 02:52:40.597670 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:40.597771 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
I20260929 02:52:41.025009 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:41.024242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:51011)
0929 02:52:41.024723 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:41.024808 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":235,"threads_started":1}
I20260929 02:52:41.435042 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:41.434137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:56225)
0929 02:52:41.434682 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:41.434783 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":266,"threads_started":1}
I20260929 02:52:41.599901 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:41.599147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.129:33623)
0929 02:52:41.599622 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:41.599720 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":233,"threads_started":1}
I20260929 02:52:42.026804 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:42.026090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.130:53617)
0929 02:52:42.026521 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:42.026625 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":215,"threads_started":1}
I20260929 02:52:42.418375 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 19863
I20260929 02:52:42.433115 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:42.419778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.4.197.129:43297)
0929 02:52:42.420323 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:42.420331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:42.420358 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:52:42.420415 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:42.420421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:42.420622 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:42.420761 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:42.422577 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:42.423686 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:42.424970 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:42.425498 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:42.425589 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:42.428118 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:42.428130 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:42.428144 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:42.428182 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:42.430731 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:42.431444 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:42.431451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:42.431454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:42.431592 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:42.431872 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:42.431875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:42.431877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:42.432197 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:42.432410 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:42.432634 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 02:52:42.432847 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":271,"threads_started":1}
I20260929 02:52:42.434419 20021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:42.437101 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:42.436188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36783 (local address 127.4.197.131:51047)
0929 02:52:42.436725 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:42.436879 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":225,"threads_started":1}
I20260929 02:52:42.520901 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 19863
I20260929 02:52:42.534770 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 20029
I20260929 02:52:42.547614 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:42.535586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.4.197.130:46797)
0929 02:52:42.536005 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:42.536010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:42.536026 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:52:42.536083 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:42.536086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:42.536145 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:42.536244 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:42.537540 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:42.538388 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:42.539677 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:42.540099 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:42.540192 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:42.542608 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:42.542613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:42.542617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:42.542652 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:42.545111 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:42.545758 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:42.545764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:42.545767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:42.545908 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:42.546199 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:42.546202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:42.546204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:42.546464 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:42.546677 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:42.547118 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 02:52:42.547323 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":224,"threads_started":1}
I20260929 02:52:42.548296 20152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:42.631209 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 20029
I20260929 02:52:42.645370 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 20160
I20260929 02:52:42.657858 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:42.646233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.4.197.131:33919)
0929 02:52:42.646778 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:42.646783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:42.646802 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:52:42.646858 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:42.646861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:42.646956 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:42.647045 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:42.648361 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:42.649135 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:42.650229 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:42.650639 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:42.650728 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:42.653302 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:42.653307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:42.653310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:42.653338 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:42.655749 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:42.656352 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:42.656355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:42.656358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:42.656478 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:42.656745 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:42.656748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:42.656750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:42.656980 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:42.657176 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:42.657381 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 02:52:42.657536 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":289,"threads_started":1}
I20260929 02:52:42.658524 20283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:42.750124 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 20160
2026-09-29T02:52:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23886 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:52:42 dist-test-slave-t6c1 krb5kdc[20353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:52:42 dist-test-slave-t6c1 krb5kdc[20353](info): set up 2 sockets
Sep 29 02:52:42 dist-test-slave-t6c1 krb5kdc[20353](info): commencing operation
krb5kdc: starting...
W20260929 02:52:43.763425 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:52:43 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:52:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:52:43.984357 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34891
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:44.312897 20366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:44.313189 20366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:44.313294 20366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:44.320784 20366 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:52:44.320876 20366 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:52:44.320922 20366 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:52:44.320972 20366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:52:44.321022 20366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:44.321087 20366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:44.321130 20366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:52:44.321180 20366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:52:44.331120 20366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:34891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:44.332360 20366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:44.333665 20366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:44.345359 20371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:44.345297 20372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:44.345329 20374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:44.346280 20366 server_base.cc:1034] running on GCE node
I20260929 02:52:44.347172 20366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:44.348631 20366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:44.349887 20366 hybrid_clock.cc:648] HybridClock initialized: now 1790650364349814 us; error 39 us; skew 500 ppm
I20260929 02:52:44.350286 20366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:44 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650364, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:44.353572 20366 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:52:44.355195 20366 webserver.cc:460] Webserver started at http://127.4.197.190:37573/ using document root <none> and password file <none>
I20260929 02:52:44.355913 20366 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:44.356016 20366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:44.356326 20366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:44.359409 20366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "b9f964978811488d919e5227bddf86a3"
format_stamp: "Formatted at 2026-09-29 02:52:44 on dist-test-slave-t6c1"
server_key: "2d4ae9da6dcec9fc1da3ab0102ba3435"
server_key_iv: "aae4c52ac425b9154ed72fe5861d843e"
server_key_version: "encryptionkey@0"
I20260929 02:52:44.360185 20366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "b9f964978811488d919e5227bddf86a3"
format_stamp: "Formatted at 2026-09-29 02:52:44 on dist-test-slave-t6c1"
server_key: "2d4ae9da6dcec9fc1da3ab0102ba3435"
server_key_iv: "aae4c52ac425b9154ed72fe5861d843e"
server_key_version: "encryptionkey@0"
I20260929 02:52:44.365638 20366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:52:44.369355 20381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:44.370882 20366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:52:44.371129 20366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "b9f964978811488d919e5227bddf86a3"
format_stamp: "Formatted at 2026-09-29 02:52:44 on dist-test-slave-t6c1"
server_key: "2d4ae9da6dcec9fc1da3ab0102ba3435"
server_key_iv: "aae4c52ac425b9154ed72fe5861d843e"
server_key_version: "encryptionkey@0"
I20260929 02:52:44.371361 20366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:44.393714 20366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:44.398000 20366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:44.398249 20366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:44.415038 20366 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:34891
I20260929 02:52:44.415050 20432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:34891 every 8 connection(s)
I20260929 02:52:44.416282 20366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:52:44.419755 20433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:44.426723 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 20366
I20260929 02:52:44.426944 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:52:44.427575 4886 external_mini_cluster.cc:1428] Setting key 0760c3f047e4e3d63789812b28901e1f
I20260929 02:52:44.430363 20433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Bootstrap starting.
I20260929 02:52:44.434515 20433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:44.435745 20433 log.cc:824] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:44.438618 20433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: No bootstrap required, opened a new log
Sep 29 02:52:44 dist-test-slave-t6c1 krb5kdc[20353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:44.447717 20433 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:52:44.448138 20433 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:44.448231 20433 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f964978811488d919e5227bddf86a3, State: Initialized, Role: FOLLOWER
I20260929 02:52:44.448926 20433 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:52:44.449227 20433 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:52:44.449370 20433 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:52:44.449546 20433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:44.452459 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:44.429628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.4.197.190:34891)
0929 02:52:44.430426 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:44.430442 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:44.430493 (+ 51us) server_negotiation.cc:407] Connection header received
0929 02:52:44.432899 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:44.432927 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:44.433451 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:44.433902 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:44.435297 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:44.437009 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:44.438354 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:44.439387 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:44.439569 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:44.443583 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:44.443594 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:44.443614 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:44.443663 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:44.447618 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:44.448348 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:44.448356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:44.448365 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:44.448591 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:44.448968 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:44.448973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:44.448975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:44.449749 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:44.450053 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:44.450448 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 02:52:44.450974 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":338,"threads_started":1}
I20260929 02:52:44.452987 20433 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:52:44.453497 20433 leader_election.cc:302] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9f964978811488d919e5227bddf86a3; no voters:
I20260929 02:52:44.454136 20433 leader_election.cc:288] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:52:44.454262 20438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:52:44.455343 20438 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 1 LEADER]: Becoming Leader. State: Replica: b9f964978811488d919e5227bddf86a3, State: Running, Role: LEADER
I20260929 02:52:44.455992 20438 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:52:44.456543 20433 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:52:44.458870 20439 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9f964978811488d919e5227bddf86a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } } }
I20260929 02:52:44.459316 20439 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:44.461338 20440 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9f964978811488d919e5227bddf86a3. Latest consensus state: current_term: 1 leader_uuid: "b9f964978811488d919e5227bddf86a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } } }
I20260929 02:52:44.461725 20440 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:52:44.462296 20444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:52:44.467123 20444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:52:44.475576 20444 catalog_manager.cc:1347] Generated new cluster ID: 732e954e031440a382e66ff71caadd6c
I20260929 02:52:44.475718 20444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:52:44.491672 20444 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:52:44.493053 20444 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:52:44.502281 20444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Generated new TSK 0
I20260929 02:52:44.503165 20444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:52:44.564894 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34891
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:44.904422 20459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:44.904776 20459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:44.904917 20459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:44.912436 20459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:44.912591 20459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:44.912757 20459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:52:44.922765 20459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:44.924199 20459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:44.925613 20459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:44.939481 20464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:44.939494 20467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:44.940318 20459 server_base.cc:1034] running on GCE node
W20260929 02:52:44.940342 20465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:44.941125 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:44.942142 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:44.943341 20459 hybrid_clock.cc:648] HybridClock initialized: now 1790650364943272 us; error 59 us; skew 500 ppm
I20260929 02:52:44.943666 20459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:44 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650364, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:44.946977 20459 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:52:44.948534 20459 webserver.cc:460] Webserver started at http://127.4.197.129:38393/ using document root <none> and password file <none>
I20260929 02:52:44.949272 20459 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:44.949410 20459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:44.949693 20459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:44.952728 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "d0878e79b7944b62bc7cf74e221cad3b"
format_stamp: "Formatted at 2026-09-29 02:52:44 on dist-test-slave-t6c1"
server_key: "362fd47f7f2e8afe8fdff3688e3a6870"
server_key_iv: "176507111dddaa0ab1cdd9d16b14f270"
server_key_version: "encryptionkey@0"
I20260929 02:52:44.953568 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "d0878e79b7944b62bc7cf74e221cad3b"
format_stamp: "Formatted at 2026-09-29 02:52:44 on dist-test-slave-t6c1"
server_key: "362fd47f7f2e8afe8fdff3688e3a6870"
server_key_iv: "176507111dddaa0ab1cdd9d16b14f270"
server_key_version: "encryptionkey@0"
I20260929 02:52:44.959270 20459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:52:44.963039 20474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:44.964468 20459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:52:44.964639 20459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "d0878e79b7944b62bc7cf74e221cad3b"
format_stamp: "Formatted at 2026-09-29 02:52:44 on dist-test-slave-t6c1"
server_key: "362fd47f7f2e8afe8fdff3688e3a6870"
server_key_iv: "176507111dddaa0ab1cdd9d16b14f270"
server_key_version: "encryptionkey@0"
I20260929 02:52:44.964850 20459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:44.989679 20459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:44.993633 20459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:44.993909 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:44.994874 20459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:44.996186 20459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:44.996269 20459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:44.996388 20459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:44.996441 20459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:45.025611 20459 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:41597
I20260929 02:52:45.025660 20586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:41597 every 8 connection(s)
I20260929 02:52:45.026961 20459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:52:45.031108 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 20459
I20260929 02:52:45.031327 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:52:45.032130 4886 external_mini_cluster.cc:1428] Setting key 1c05fe555504a0d4a5f5d942a410425a
Sep 29 02:52:45 dist-test-slave-t6c1 krb5kdc[20353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650364, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:45.049909 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:45.029524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:39821 (local address 127.4.197.190:34891)
0929 02:52:45.029999 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:45.030006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:45.031165 (+ 1159us) server_negotiation.cc:407] Connection header received
0929 02:52:45.033626 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:45.033630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:45.033709 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:45.033797 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:45.035984 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:45.036717 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:45.038892 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:45.039372 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:45.039479 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:45.043850 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:45.043856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:45.043860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:45.043900 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:45.046490 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:45.047428 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:45.047433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:45.047436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:45.047575 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:45.048043 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:45.048048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:45.048050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:45.048343 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:45.048488 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:45.049277 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 02:52:45.049529 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":215,"threads_started":1}
I20260929 02:52:45.050987 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:45.030138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:39821)
0929 02:52:45.030974 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:45.031033 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:45.033267 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:45.034096 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:45.034106 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:45.034888 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:45.035765 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:45.035793 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:45.036890 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:45.036894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:45.038715 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:45.038730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:45.039602 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:45.039609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:45.039913 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:45.040636 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:45.040665 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:45.043573 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:45.046675 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:45.046685 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:45.046702 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:45.047283 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:45.047692 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:45.047697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:45.047701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:45.047916 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:45.048488 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:45.048497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:45.048874 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 02:52:45.049131 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 02:52:45.049733 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":272,"threads_started":1}
I20260929 02:52:45.052661 20587 heartbeater.cc:347] Connected to a master server at 127.4.197.190:34891
I20260929 02:52:45.053025 20587 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:45.053941 20587 heartbeater.cc:511] Master 127.4.197.190:34891 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:52:45.056206 20398 ts_manager.cc:194] Registered new tserver with Master: d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597)
I20260929 02:52:45.057777 20398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:39821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:52:45.086512 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34891
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:45.410179 20593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:45.410461 20593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:45.410593 20593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:45.417841 20593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:45.417937 20593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:45.418066 20593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:52:45.427907 20593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:45.429220 20593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:45.430601 20593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:45.443116 20601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:45.443184 20599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:45.443159 20598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:45.443861 20593 server_base.cc:1034] running on GCE node
I20260929 02:52:45.444427 20593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:45.445318 20593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:45.446518 20593 hybrid_clock.cc:648] HybridClock initialized: now 1790650365446469 us; error 35 us; skew 500 ppm
I20260929 02:52:45.446825 20593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:45 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650365, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:45.449699 20593 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:52:45.451116 20593 webserver.cc:460] Webserver started at http://127.4.197.130:42077/ using document root <none> and password file <none>
I20260929 02:52:45.451757 20593 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:45.451869 20593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:45.452140 20593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:45.454867 20593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "9ac614ef30c048558e64976e05863a4d"
format_stamp: "Formatted at 2026-09-29 02:52:45 on dist-test-slave-t6c1"
server_key: "d9869e03b78c2639426a5413b2197861"
server_key_iv: "dc8a3a4e60eb6a63853a78991e4129c7"
server_key_version: "encryptionkey@0"
I20260929 02:52:45.455598 20593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "9ac614ef30c048558e64976e05863a4d"
format_stamp: "Formatted at 2026-09-29 02:52:45 on dist-test-slave-t6c1"
server_key: "d9869e03b78c2639426a5413b2197861"
server_key_iv: "dc8a3a4e60eb6a63853a78991e4129c7"
server_key_version: "encryptionkey@0"
I20260929 02:52:45.460644 20593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:52:45.464480 20608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:45.465969 20593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:52:45.466148 20593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "9ac614ef30c048558e64976e05863a4d"
format_stamp: "Formatted at 2026-09-29 02:52:45 on dist-test-slave-t6c1"
server_key: "d9869e03b78c2639426a5413b2197861"
server_key_iv: "dc8a3a4e60eb6a63853a78991e4129c7"
server_key_version: "encryptionkey@0"
I20260929 02:52:45.466355 20593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:45.484377 20593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:45.488590 20593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:45.488857 20593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:45.489857 20593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:45.491255 20593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:45.491346 20593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:45.491465 20593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:45.491518 20593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:45.521862 20593 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:43449
I20260929 02:52:45.521910 20720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:43449 every 8 connection(s)
I20260929 02:52:45.523242 20593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:52:45.529218 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 20593
I20260929 02:52:45.529430 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:52:45.530055 4886 external_mini_cluster.cc:1428] Setting key f3acb4299da60c1368407e399833524b
Sep 29 02:52:45 dist-test-slave-t6c1 krb5kdc[20353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650365, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:45.544982 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:45.525687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:33071 (local address 127.4.197.190:34891)
0929 02:52:45.525846 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:45.525850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:45.527051 (+ 1201us) server_negotiation.cc:407] Connection header received
0929 02:52:45.528946 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:45.528951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:45.529027 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:45.529117 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:45.531132 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:45.531897 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:45.534091 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:45.534608 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:45.534732 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:45.538827 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:45.538833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:45.538837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:45.538873 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:45.541783 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:45.542646 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:45.542651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:45.542654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:45.542807 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:45.543275 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:45.543279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:45.543282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:45.543624 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:45.543762 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:45.544443 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 02:52:45.544669 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:52:45.545907 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:45.526168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:33071)
0929 02:52:45.526872 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:45.526911 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:45.528663 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:45.529424 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:45.529436 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:45.530162 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:45.530946 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:45.530973 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:45.532042 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:45.532047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:45.533936 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:45.533953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:45.534833 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:45.534840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:45.535148 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:45.535891 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:45.535918 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:45.538526 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:45.541991 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:45.542002 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:45.542022 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:45.542502 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:45.542954 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:45.542958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:45.542962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:45.543164 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:45.543732 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:45.543738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:45.544092 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 02:52:45.544320 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 02:52:45.544741 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":270,"threads_started":1}
I20260929 02:52:45.547613 20721 heartbeater.cc:347] Connected to a master server at 127.4.197.190:34891
I20260929 02:52:45.547921 20721 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:45.548659 20721 heartbeater.cc:511] Master 127.4.197.190:34891 requested a full tablet report, sending...
I20260929 02:52:45.550149 20398 ts_manager.cc:194] Registered new tserver with Master: 9ac614ef30c048558e64976e05863a4d (127.4.197.130:43449)
I20260929 02:52:45.551005 20398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:33071
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:52:45.581895 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34891
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:52:45.905177 20726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:52:45.905478 20726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:52:45.905593 20726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:52:45.912959 20726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:52:45.913080 20726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:52:45.913233 20726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:52:45.922973 20726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:52:45.924263 20726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:52:45.925571 20726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:52:45.938665 20732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:45.938730 20731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:52:45.938766 20734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:52:45.939596 20726 server_base.cc:1034] running on GCE node
I20260929 02:52:45.940256 20726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:52:45.941355 20726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:52:45.942590 20726 hybrid_clock.cc:648] HybridClock initialized: now 1790650365942543 us; error 38 us; skew 500 ppm
I20260929 02:52:45.942993 20726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:52:45 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650365, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:52:45.957728 20726 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:52:45.959558 20726 webserver.cc:460] Webserver started at http://127.4.197.131:42255/ using document root <none> and password file <none>
I20260929 02:52:45.960371 20726 fs_manager.cc:359] Metadata directory not provided
I20260929 02:52:45.960501 20726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:52:45.960846 20726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:52:45.964227 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "4f8a55e7c21746629a6757313665b2ae"
format_stamp: "Formatted at 2026-09-29 02:52:45 on dist-test-slave-t6c1"
server_key: "7304392a848a5b466b20ed1df81b4aa0"
server_key_iv: "1444f133703bf2f580acfbf311c76753"
server_key_version: "encryptionkey@0"
I20260929 02:52:45.965091 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "4f8a55e7c21746629a6757313665b2ae"
format_stamp: "Formatted at 2026-09-29 02:52:45 on dist-test-slave-t6c1"
server_key: "7304392a848a5b466b20ed1df81b4aa0"
server_key_iv: "1444f133703bf2f580acfbf311c76753"
server_key_version: "encryptionkey@0"
I20260929 02:52:45.970706 20726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:52:45.974332 20741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:45.975934 20726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:52:45.976102 20726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "4f8a55e7c21746629a6757313665b2ae"
format_stamp: "Formatted at 2026-09-29 02:52:45 on dist-test-slave-t6c1"
server_key: "7304392a848a5b466b20ed1df81b4aa0"
server_key_iv: "1444f133703bf2f580acfbf311c76753"
server_key_version: "encryptionkey@0"
I20260929 02:52:45.976293 20726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:52:45.995849 20726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:52:46.000594 20726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:52:46.000892 20726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:52:46.001997 20726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:52:46.003584 20726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:52:46.003669 20726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:46.003798 20726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:52:46.003850 20726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:52:46.034332 20726 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:39235
I20260929 02:52:46.034368 20853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:39235 every 8 connection(s)
I20260929 02:52:46.035674 20726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:52:46.044062 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 20726
I20260929 02:52:46.044256 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:52:46.044699 4886 external_mini_cluster.cc:1428] Setting key 592e1300aea0716c410ac737d231608a
Sep 29 02:52:46 dist-test-slave-t6c1 krb5kdc[20353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650365, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:52:46.056097 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.038114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:45743 (local address 127.4.197.190:34891)
0929 02:52:46.038296 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.038301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.039630 (+ 1329us) server_negotiation.cc:407] Connection header received
0929 02:52:46.041460 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.041466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.041550 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.041636 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:46.043284 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.044292 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.046212 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.046794 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.046940 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.050574 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:46.050580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:46.050583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:46.050620 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:46.053227 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:46.053987 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:46.053991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:46.053993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:46.054123 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:46.054525 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:46.054528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:46.054530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:46.054795 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:46.054913 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.055649 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.055827 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:52:46.057221 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.038712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:45743)
0929 02:52:46.039458 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.039502 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:46.041165 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:46.041905 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:46.041914 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:46.042437 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:52:46.043075 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.043094 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.044460 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.044464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.046045 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.046061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.047090 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.047098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.047412 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.048147 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:52:46.048176 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:52:46.050375 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:52:46.053375 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:46.053383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:46.053396 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:46.053887 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:46.054212 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:52:46.054216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:52:46.054218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:52:46.054425 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:52:46.054969 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:52:46.054977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:52:46.055324 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 02:52:46.055571 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 02:52:46.056003 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":306,"threads_started":1}
I20260929 02:52:46.058789 20854 heartbeater.cc:347] Connected to a master server at 127.4.197.190:34891
I20260929 02:52:46.059094 20854 heartbeater.cc:464] Registering TS with master...
I20260929 02:52:46.059898 20854 heartbeater.cc:511] Master 127.4.197.190:34891 requested a full tablet report, sending...
I20260929 02:52:46.061287 20587 heartbeater.cc:503] Master 127.4.197.190:34891 was elected leader, sending a full tablet report...
I20260929 02:52:46.061297 20398 ts_manager.cc:194] Registered new tserver with Master: 4f8a55e7c21746629a6757313665b2ae (127.4.197.131:39235)
I20260929 02:52:46.062399 20398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:45743
I20260929 02:52:46.073038 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:52:46.090301 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.077793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.4.197.190:34891)
0929 02:52:46.078033 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.078040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.078299 (+ 259us) server_negotiation.cc:407] Connection header received
0929 02:52:46.078451 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.078455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.078520 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.078601 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:46.080349 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.081189 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.082559 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.083027 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.083116 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.085431 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:46.085436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:46.085439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:46.085472 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:46.087921 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:46.088600 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:46.088604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:46.088606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:46.088744 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:46.089061 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:46.089064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:46.089067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:46.089394 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:46.089575 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.089817 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.090001 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 02:52:46.094264 20398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:52:46.101032 20398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:52:46.127571 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.112614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:39235 (local address 127.0.0.1:32802)
0929 02:52:46.113309 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.113360 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:46.113552 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:46.120500 (+ 6948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:46.120508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:46.120540 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:46.121033 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.121045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.122660 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.122664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.124841 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.124853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.126478 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.126484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.126681 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.126734 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 02:52:46.126857 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 02:52:46.127070 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":2,"thread_start_us":762,"threads_started":1}
I20260929 02:52:46.128284 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.113358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.0.0.1:60904)
0929 02:52:46.120117 (+ 6759us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.120134 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:46.120353 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:46.121265 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:46.121269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:46.121292 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:46.121672 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.121684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.123544 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.123549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.125690 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.125701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.127607 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.127613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.127772 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.127800 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:52:46.127885 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 02:52:46.128001 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6641,"mutex_wait_us":30,"thread_start_us":6507,"threads_started":1}
I20260929 02:52:46.128516 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.112783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.4.197.131:39235)
0929 02:52:46.119821 (+ 7038us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.119830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.119862 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:52:46.119948 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.119955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.120239 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.121052 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:46.121323 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.122530 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.124996 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.126279 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.126418 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.127958 (+ 1540us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.128030 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.128190 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6851,"thread_start_us":6681,"threads_started":1}
I20260929 02:52:46.128643 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.113989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.4.197.130:43449)
0929 02:52:46.114636 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.114647 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.119559 (+ 4912us) server_negotiation.cc:407] Connection header received
0929 02:52:46.119946 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.119955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.120239 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.120434 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:46.121485 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.122975 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.125551 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.127022 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.127153 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.127346 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.127895 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.128346 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":735,"threads_started":1}
I20260929 02:52:46.129153 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.113052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:43449 (local address 127.0.0.1:38332)
0929 02:52:46.119412 (+ 6360us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.119434 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:46.119690 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:46.120737 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:46.120741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:46.120762 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:46.121289 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.121304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.123136 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.123141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.125375 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.125389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.127514 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.127519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.127695 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.127724 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:52:46.128728 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0929 02:52:46.128868 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6218,"mutex_wait_us":64,"thread_start_us":6088,"threads_started":1}
I20260929 02:52:46.129668 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.114512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.4.197.129:41597)
0929 02:52:46.115263 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.115272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.120244 (+ 4972us) server_negotiation.cc:407] Connection header received
0929 02:52:46.120549 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.120558 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.120975 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.121146 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:46.121843 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.123378 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.125844 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.127117 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.127243 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.127457 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.128535 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.129393 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":433,"threads_started":1}
I20260929 02:52:46.132503 20789 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa5f7068ecf4a8095bf66d247b7ff92 (DEFAULT_TABLE table=test-table [id=e125aa5beb1a496c9c1401bd655461f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:46.132550 20656 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa5f7068ecf4a8095bf66d247b7ff92 (DEFAULT_TABLE table=test-table [id=e125aa5beb1a496c9c1401bd655461f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:46.133028 20522 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa5f7068ecf4a8095bf66d247b7ff92 (DEFAULT_TABLE table=test-table [id=e125aa5beb1a496c9c1401bd655461f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:52:46.133920 20789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa5f7068ecf4a8095bf66d247b7ff92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:46.133991 20656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa5f7068ecf4a8095bf66d247b7ff92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:46.134236 20522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa5f7068ecf4a8095bf66d247b7ff92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:52:46.144773 20872 tablet_bootstrap.cc:492] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: Bootstrap starting.
I20260929 02:52:46.147825 20872 tablet_bootstrap.cc:654] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:46.147866 20874 tablet_bootstrap.cc:492] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: Bootstrap starting.
I20260929 02:52:46.148654 20873 tablet_bootstrap.cc:492] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Bootstrap starting.
I20260929 02:52:46.149940 20872 log.cc:824] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:46.151022 20874 tablet_bootstrap.cc:654] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:46.151661 20873 tablet_bootstrap.cc:654] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Neither blocks nor log segments found. Creating new log.
I20260929 02:52:46.152143 20874 log.cc:824] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:46.152560 20873 log.cc:824] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:52:46.152979 20872 tablet_bootstrap.cc:492] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: No bootstrap required, opened a new log
I20260929 02:52:46.153229 20872 ts_tablet_manager.cc:1397] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:52:46.154981 20874 tablet_bootstrap.cc:492] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: No bootstrap required, opened a new log
I20260929 02:52:46.155221 20874 ts_tablet_manager.cc:1397] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 02:52:46.156558 20873 tablet_bootstrap.cc:492] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: No bootstrap required, opened a new log
I20260929 02:52:46.156780 20873 ts_tablet_manager.cc:1397] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 02:52:46.164177 20873 raft_consensus.cc:348] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:52:46.164180 20872 raft_consensus.cc:348] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:52:46.164599 20873 raft_consensus.cc:374] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:46.164644 20872 raft_consensus.cc:374] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:46.164726 20873 raft_consensus.cc:729] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0878e79b7944b62bc7cf74e221cad3b, State: Initialized, Role: FOLLOWER
I20260929 02:52:46.164777 20872 raft_consensus.cc:729] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8a55e7c21746629a6757313665b2ae, State: Initialized, Role: FOLLOWER
I20260929 02:52:46.165400 20873 consensus_queue.cc:260] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:52:46.165516 20872 consensus_queue.cc:260] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:52:46.165509 20874 raft_consensus.cc:348] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:52:46.166044 20874 raft_consensus.cc:374] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:52:46.166204 20874 raft_consensus.cc:729] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac614ef30c048558e64976e05863a4d, State: Initialized, Role: FOLLOWER
I20260929 02:52:46.166999 20874 consensus_queue.cc:260] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:52:46.168512 20873 ts_tablet_manager.cc:1428] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 02:52:46.169637 20854 heartbeater.cc:503] Master 127.4.197.190:34891 was elected leader, sending a full tablet report...
I20260929 02:52:46.170275 20872 ts_tablet_manager.cc:1428] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: Time spent starting tablet: real 0.017s user 0.001s sys 0.014s
I20260929 02:52:46.171573 20721 heartbeater.cc:503] Master 127.4.197.190:34891 was elected leader, sending a full tablet report...
I20260929 02:52:46.172161 20874 ts_tablet_manager.cc:1428] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: Time spent starting tablet: real 0.017s user 0.009s sys 0.005s
I20260929 02:52:46.255551 20880 raft_consensus.cc:482] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:46.255836 20880 raft_consensus.cc:504] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:52:46.257592 20880 leader_election.cc:288] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8a55e7c21746629a6757313665b2ae (127.4.197.131:39235), d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597)
I20260929 02:52:46.261811 20879 raft_consensus.cc:482] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:52:46.262152 20879 raft_consensus.cc:504] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:52:46.264408 20879 leader_election.cc:288] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597), 9ac614ef30c048558e64976e05863a4d (127.4.197.130:43449)
I20260929 02:52:46.264621 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.257900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:59249 (local address 127.4.197.131:39235)
0929 02:52:46.258104 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.258112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.258781 (+ 669us) server_negotiation.cc:407] Connection header received
0929 02:52:46.259004 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.259008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.259069 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.259143 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:46.259685 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.260451 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.262650 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.263675 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.263829 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.263917 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.264191 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.264316 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:52:46.264732 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.257877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:39235 (local address 127.4.197.130:59249)
0929 02:52:46.258658 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.258673 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:46.258844 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:46.259172 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:46.259177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:46.259202 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:46.259536 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.259545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.260578 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.260583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.262435 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.262457 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.263883 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.263890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.264043 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.264068 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:52:46.264150 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 02:52:46.264290 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":1,"thread_start_us":395,"threads_started":1}
I20260929 02:52:46.265297 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.258241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:34657 (local address 127.4.197.129:41597)
0929 02:52:46.258466 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.258472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.259080 (+ 608us) server_negotiation.cc:407] Connection header received
0929 02:52:46.259240 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.259244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.259303 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.259376 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:46.259938 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.260712 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.263000 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.264170 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.264330 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.264417 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.264937 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.265067 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:52:46.265398 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.257881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.4.197.130:34657)
0929 02:52:46.258967 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.258984 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:46.259121 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:46.259419 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:46.259423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:46.259446 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:46.259773 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.259781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.260841 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.260847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.262805 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.262819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.264580 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.264586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.264779 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.264813 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:52:46.264899 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 02:52:46.265022 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":124,"thread_start_us":697,"threads_started":1}
I20260929 02:52:46.266167 20809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa5f7068ecf4a8095bf66d247b7ff92" candidate_uuid: "9ac614ef30c048558e64976e05863a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8a55e7c21746629a6757313665b2ae" is_pre_election: true
I20260929 02:52:46.266681 20809 raft_consensus.cc:2454] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ac614ef30c048558e64976e05863a4d in term 0.
I20260929 02:52:46.266667 20542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa5f7068ecf4a8095bf66d247b7ff92" candidate_uuid: "9ac614ef30c048558e64976e05863a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0878e79b7944b62bc7cf74e221cad3b" is_pre_election: true
I20260929 02:52:46.267376 20542 raft_consensus.cc:2454] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ac614ef30c048558e64976e05863a4d in term 0.
I20260929 02:52:46.267407 20609 leader_election.cc:302] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8a55e7c21746629a6757313665b2ae, 9ac614ef30c048558e64976e05863a4d; no voters:
I20260929 02:52:46.267856 20880 raft_consensus.cc:2781] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:52:46.268003 20880 raft_consensus.cc:482] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:52:46.268113 20880 raft_consensus.cc:3037] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:46.271406 20880 raft_consensus.cc:504] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:52:46.272179 20880 leader_election.cc:288] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [CANDIDATE]: Term 1 election: Requested vote from peers 4f8a55e7c21746629a6757313665b2ae (127.4.197.131:39235), d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597)
I20260929 02:52:46.272603 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.264904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.4.197.131:45927)
0929 02:52:46.265157 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.265175 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:46.265282 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:46.266174 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:46.266178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:46.266205 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:46.266412 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.266424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.268021 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.268027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.269880 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.269897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.271640 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.271648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.271949 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.271997 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 02:52:46.272107 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 02:52:46.272273 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260929 02:52:46.273216 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.265513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:45927 (local address 127.4.197.129:41597)
0929 02:52:46.265885 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.265891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.265909 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:52:46.265969 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.265974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.266046 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.266549 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:46.266703 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.267837 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.270114 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.271423 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.271613 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.271683 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.272791 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.272938 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"threads_started":1}
I20260929 02:52:46.273130 20809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa5f7068ecf4a8095bf66d247b7ff92" candidate_uuid: "9ac614ef30c048558e64976e05863a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8a55e7c21746629a6757313665b2ae"
I20260929 02:52:46.273281 20542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa5f7068ecf4a8095bf66d247b7ff92" candidate_uuid: "9ac614ef30c048558e64976e05863a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0878e79b7944b62bc7cf74e221cad3b"
I20260929 02:52:46.273379 20809 raft_consensus.cc:3037] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:52:46.273521 20542 raft_consensus.cc:3037] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:52:46.275166 20722 tablet.cc:2369] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:52:46.276358 20809 raft_consensus.cc:2454] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac614ef30c048558e64976e05863a4d in term 1.
I20260929 02:52:46.276772 20609 leader_election.cc:302] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8a55e7c21746629a6757313665b2ae, 9ac614ef30c048558e64976e05863a4d; no voters:
I20260929 02:52:46.277312 20542 raft_consensus.cc:2454] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac614ef30c048558e64976e05863a4d in term 1.
I20260929 02:52:46.277673 20880 raft_consensus.cc:2781] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 1 FOLLOWER]: Leader election won for term 1
W20260929 02:52:46.279327 20588 tablet.cc:2369] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:52:46.279680 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.266120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:55421 (local address 127.4.197.130:43449)
0929 02:52:46.267959 (+ 1839us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.267965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.272309 (+ 4344us) server_negotiation.cc:407] Connection header received
0929 02:52:46.272572 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.272578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.272652 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.272711 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:52:46.273522 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.274515 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.277139 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.278376 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.278517 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.278596 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.279230 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.279382 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1694}
I20260929 02:52:46.280380 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.265510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:43449 (local address 127.4.197.131:55421)
0929 02:52:46.272210 (+ 6700us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.272232 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:52:46.272421 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:52:46.272921 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:52:46.272926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:52:46.272954 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:52:46.273328 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.273341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.274731 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.274736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.276949 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:52:46.276965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.278794 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.278800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:52:46.279029 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.279068 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:52:46.279845 (+ 777us) client_negotiation.cc:240] Negotiation successful
0929 02:52:46.280027 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6550,"thread_start_us":10581,"threads_started":1}
I20260929 02:52:46.280905 20880 raft_consensus.cc:686] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 1 LEADER]: Becoming Leader. State: Replica: 9ac614ef30c048558e64976e05863a4d, State: Running, Role: LEADER
I20260929 02:52:46.281553 20880 consensus_queue.cc:237] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:52:46.281699 20676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa5f7068ecf4a8095bf66d247b7ff92" candidate_uuid: "4f8a55e7c21746629a6757313665b2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac614ef30c048558e64976e05863a4d" is_pre_election: true
I20260929 02:52:46.284827 20542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa5f7068ecf4a8095bf66d247b7ff92" candidate_uuid: "4f8a55e7c21746629a6757313665b2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0878e79b7944b62bc7cf74e221cad3b" is_pre_election: true
I20260929 02:52:46.285158 20542 raft_consensus.cc:2379] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4f8a55e7c21746629a6757313665b2ae in current term 1: Already voted for candidate 9ac614ef30c048558e64976e05863a4d in this term.
I20260929 02:52:46.285799 20743 leader_election.cc:302] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f8a55e7c21746629a6757313665b2ae; no voters: 9ac614ef30c048558e64976e05863a4d, d0878e79b7944b62bc7cf74e221cad3b
I20260929 02:52:46.286221 20879 raft_consensus.cc:2726] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 02:52:46.287122 20855 tablet.cc:2369] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:52:46.286713 20396 catalog_manager.cc:5495] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ac614ef30c048558e64976e05863a4d (127.4.197.130). New cstate: current_term: 1 leader_uuid: "9ac614ef30c048558e64976e05863a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } health_report { overall_health: HEALTHY } } }
I20260929 02:52:46.326148 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.321223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.4.197.130:43449)
0929 02:52:46.321428 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.321432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.321449 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:52:46.321632 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.321636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.321729 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.321843 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:52:46.322126 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.322837 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.324250 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.324645 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.324732 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.325144 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:52:46.325627 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:52:46.325736 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.325777 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.325844 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:52:46.334216 20809 raft_consensus.cc:1261] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 1 FOLLOWER]: Refusing update from remote peer 9ac614ef30c048558e64976e05863a4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:52:46.334177 20542 raft_consensus.cc:1261] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Refusing update from remote peer 9ac614ef30c048558e64976e05863a4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:52:46.335165 20880 consensus_queue.cc:1035] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:52:46.335453 20885 consensus_queue.cc:1035] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:52:46.361214 20396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:52:46.361490 20396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:52:46.365056 20396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Sending DeleteTablet for 3 replicas of tablet 6fa5f7068ecf4a8095bf66d247b7ff92
I20260929 02:52:46.366086 20522 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa5f7068ecf4a8095bf66d247b7ff92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:46 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:60904
I20260929 02:52:46.366091 20656 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa5f7068ecf4a8095bf66d247b7ff92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:46 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:38332
I20260929 02:52:46.366117 20789 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa5f7068ecf4a8095bf66d247b7ff92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:46 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:32802
I20260929 02:52:46.368590 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 20459
I20260929 02:52:46.383512 20898 tablet_replica.cc:330] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: stopping tablet replica
I20260929 02:52:46.384107 20899 tablet_replica.cc:330] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: stopping tablet replica
I20260929 02:52:46.384801 20899 raft_consensus.cc:2229] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:52:46.385402 20898 raft_consensus.cc:2229] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:52:46.386047 20899 raft_consensus.cc:2258] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:46.386962 20900 tablet_replica.cc:330] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: stopping tablet replica
I20260929 02:52:46.387127 20898 raft_consensus.cc:2258] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:46.387470 20900 raft_consensus.cc:2229] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:52:46.388027 20900 raft_consensus.cc:2258] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:52:46.389670 20898 ts_tablet_manager.cc:1905] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:52:46.390239 20900 ts_tablet_manager.cc:1905] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:52:46 dist-test-slave-t6c1 krb5kdc[20353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:52:46.396970 20900 ts_tablet_manager.cc:1918] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:52:46.396977 20898 ts_tablet_manager.cc:1918] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:52:46.397137 20898 log.cc:1196] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/6fa5f7068ecf4a8095bf66d247b7ff92
I20260929 02:52:46.397217 20900 log.cc:1196] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/6fa5f7068ecf4a8095bf66d247b7ff92
I20260929 02:52:46.397709 20898 ts_tablet_manager.cc:1939] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 9ac614ef30c048558e64976e05863a4d: Deleting consensus metadata
I20260929 02:52:46.397750 20900 ts_tablet_manager.cc:1939] T 6fa5f7068ecf4a8095bf66d247b7ff92 P 4f8a55e7c21746629a6757313665b2ae: Deleting consensus metadata
I20260929 02:52:46.399494 20385 catalog_manager.cc:4841] TS 9ac614ef30c048558e64976e05863a4d (127.4.197.130:43449): tablet 6fa5f7068ecf4a8095bf66d247b7ff92 (table test-table [id=e125aa5beb1a496c9c1401bd655461f2]) successfully deleted
I20260929 02:52:46.399613 20382 catalog_manager.cc:4841] TS 4f8a55e7c21746629a6757313665b2ae (127.4.197.131:39235): tablet 6fa5f7068ecf4a8095bf66d247b7ff92 (table test-table [id=e125aa5beb1a496c9c1401bd655461f2]) successfully deleted
I20260929 02:52:46.399886 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.369378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.4.197.129:41597)
0929 02:52:46.369590 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.369596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.375680 (+ 6084us) server_negotiation.cc:407] Connection header received
0929 02:52:46.376667 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.376673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.376763 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.376855 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:46.381377 (+ 4522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.382681 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.384927 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.385524 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.385627 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.392166 (+ 6539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:46.392185 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:46.392203 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:46.392270 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:46.396062 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:46.396993 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:46.397003 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:46.397007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:46.397166 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:46.397642 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:46.397647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:46.397651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:46.398062 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:46.398365 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.399269 (+ 904us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.399504 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:52:46.400269 20899 ts_tablet_manager.cc:1905] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:52:46.401346 20582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:46.504320 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 20459
I20260929 02:52:46.518977 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 20593
W20260929 02:52:46.519016 20383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.197.129:41597 (error 108)
I20260929 02:52:46.521482 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.520993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.0.0.1:60928)
0929 02:52:46.521177 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.521292 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 02:52:46.521894 20383 catalog_manager.cc:4584] TS d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fa5f7068ecf4a8095bf66d247b7ff92: Network error: Client connection negotiation failed: client connection to 127.4.197.129:41597: connect: Connection refused (error 111)
Sep 29 02:52:46 dist-test-slave-t6c1 krb5kdc[20353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:52:46.539399 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.520039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.4.197.130:43449)
0929 02:52:46.520256 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.520261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.520276 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:52:46.520330 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.520334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.520392 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.520479 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:46.522386 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.523660 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.525692 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.526384 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.526508 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.531942 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:46.531957 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:46.531979 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:46.532023 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:46.536136 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:46.537034 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:46.537044 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:46.537048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:46.537227 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:46.537629 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:46.537633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:46.537636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:46.538172 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:46.538474 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.538777 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.539048 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:52:46.540102 20716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:46.575452 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.574965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.0.0.1:60940)
0929 02:52:46.575161 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.575265 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 02:52:46.576514 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.576128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.0.0.1:60946)
0929 02:52:46.576275 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.576350 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 02:52:46.636914 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.636409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.0.0.1:60960)
0929 02:52:46.636632 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.636736 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 02:52:46.637986 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.637616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.0.0.1:60976)
0929 02:52:46.637768 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.637843 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:52:46.653604 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 20593
I20260929 02:52:46.668162 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 20726
Sep 29 02:52:46 dist-test-slave-t6c1 krb5kdc[20353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:52:46.683826 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.668928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.4.197.131:39235)
0929 02:52:46.669112 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:52:46.669118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:52:46.669136 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:52:46.669258 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:52:46.669263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:52:46.669335 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:52:46.669424 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:52:46.670705 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.671517 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.672964 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:52:46.673367 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:52:46.673479 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:52:46.678158 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:52:46.678175 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:52:46.678194 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:52:46.678233 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:52:46.680973 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:46.681694 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:46.681702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:46.681704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:46.681861 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:52:46.682239 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:52:46.682243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:52:46.682245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:52:46.682631 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:52:46.683084 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0929 02:52:46.683248 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 02:52:46.683475 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:52:46.684471 20849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:46.718459 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.717990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.0.0.1:60980)
0929 02:52:46.718196 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.718286 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 02:52:46.719457 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.719095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.0.0.1:60988)
0929 02:52:46.719282 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.719351 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 02:52:46.773689 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 20726
I20260929 02:52:46.786734 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 20366
I20260929 02:52:46.787878 20428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:52:46.896888 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:52:46.896274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.0.0.1:60994)
0929 02:52:46.896500 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 02:52:46.896651 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 02:52:46.897081 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 20366
I20260929 02:53:06.914860 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34891
--webserver_interface=127.4.197.190
--webserver_port=37573
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:53:07.235136 20907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:07.235476 20907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:07.235605 20907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:07.242879 20907 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:53:07.243036 20907 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:53:07.243094 20907 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:53:07.243165 20907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:53:07.243248 20907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:07.243346 20907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:07.243408 20907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:53:07.243481 20907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:53:07.253652 20907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:34891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=37573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:07.255414 20907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:07.257251 20907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:07.280260 20916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:07.280233 20914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:07.280342 20913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:07.281031 20907 server_base.cc:1034] running on GCE node
I20260929 02:53:07.281952 20907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:07.283609 20907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:07.284842 20907 hybrid_clock.cc:648] HybridClock initialized: now 1790650387284759 us; error 52 us; skew 500 ppm
I20260929 02:53:07.285290 20907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:07 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:07.292289 20907 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:53:07.294034 20907 webserver.cc:460] Webserver started at http://127.4.197.190:37573/ using document root <none> and password file <none>
I20260929 02:53:07.294854 20907 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:07.295017 20907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:07.301568 20907 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 02:53:07.304694 20923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:07.306131 20907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 02:53:07.306341 20907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "b9f964978811488d919e5227bddf86a3"
format_stamp: "Formatted at 2026-09-29 02:52:44 on dist-test-slave-t6c1"
server_key: "2d4ae9da6dcec9fc1da3ab0102ba3435"
server_key_iv: "aae4c52ac425b9154ed72fe5861d843e"
server_key_version: "encryptionkey@0"
I20260929 02:53:07.307436 20907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:07.339946 20907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:07.344163 20907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:07.344455 20907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:07.361930 20907 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:34891
I20260929 02:53:07.361953 20974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:34891 every 8 connection(s)
I20260929 02:53:07.363440 20907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:53:07.367175 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 20907
I20260929 02:53:07.368108 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:41597
--local_ip_for_outbound_sockets=127.4.197.129
--tserver_master_addrs=127.4.197.190:34891
--webserver_port=38393
--webserver_interface=127.4.197.129
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:53:07.371515 20975 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:53:07.375875 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Bootstrap starting.
I20260929 02:53:07.395797 20975 log.cc:824] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:07.407464 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:53:07.407931 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Bootstrap complete.
I20260929 02:53:07.417011 20975 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:53:07.417846 20975 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f964978811488d919e5227bddf86a3, State: Initialized, Role: FOLLOWER
I20260929 02:53:07.418454 20975 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:53:07.418690 20975 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:53:07.418798 20975 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:53:07.418977 20975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:53:07.422698 20975 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:53:07.423229 20975 leader_election.cc:302] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9f964978811488d919e5227bddf86a3; no voters:
I20260929 02:53:07.423859 20975 leader_election.cc:288] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:53:07.424010 20979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:53:07.425189 20979 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 2 LEADER]: Becoming Leader. State: Replica: b9f964978811488d919e5227bddf86a3, State: Running, Role: LEADER
I20260929 02:53:07.425709 20979 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:53:07.426087 20975 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:53:07.428243 20981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9f964978811488d919e5227bddf86a3. Latest consensus state: current_term: 2 leader_uuid: "b9f964978811488d919e5227bddf86a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } } }
I20260929 02:53:07.428256 20980 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b9f964978811488d919e5227bddf86a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } } }
I20260929 02:53:07.428651 20981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:07.428681 20980 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:07.440410 20985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:53:07.449848 20985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:53:07.451449 20985 catalog_manager.cc:1259] Loaded cluster ID: 732e954e031440a382e66ff71caadd6c
I20260929 02:53:07.451622 20985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:53:07.456048 20985 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:53:07.457759 20985 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Loaded TSK: 0
I20260929 02:53:07.489915 20985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Generated new TSK 1
I20260929 02:53:07.490311 20985 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:53:07.745245 20977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:07.745720 20977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:07.745864 20977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:07.753329 20977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:07.753466 20977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:07.753616 20977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:53:07.763697 20977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:41597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=38393
--tserver_master_addrs=127.4.197.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:07.765046 20977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:07.766377 20977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:07.781572 21002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:07.781617 21003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:07.781701 21005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:07.782828 20977 server_base.cc:1034] running on GCE node
I20260929 02:53:07.783504 20977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:07.784489 20977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:07.785687 20977 hybrid_clock.cc:648] HybridClock initialized: now 1790650387785617 us; error 64 us; skew 500 ppm
I20260929 02:53:07.785998 20977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:07 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:07.789078 20977 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:53:07.790573 20977 webserver.cc:460] Webserver started at http://127.4.197.129:38393/ using document root <none> and password file <none>
I20260929 02:53:07.791288 20977 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:07.791419 20977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:07.796627 20977 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 02:53:07.799350 21012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:07.800614 20977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:53:07.800751 20977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "d0878e79b7944b62bc7cf74e221cad3b"
format_stamp: "Formatted at 2026-09-29 02:52:44 on dist-test-slave-t6c1"
server_key: "362fd47f7f2e8afe8fdff3688e3a6870"
server_key_iv: "176507111dddaa0ab1cdd9d16b14f270"
server_key_version: "encryptionkey@0"
I20260929 02:53:07.801617 20977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:07.817636 20977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:07.821473 20977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:07.821697 20977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:07.822638 20977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:07.824415 21019 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 02:53:07.829944 20977 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 02:53:07.830054 20977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 02:53:07.830173 20977 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 02:53:07.833292 20977 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 02:53:07.833389 20977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260929 02:53:07.833753 21019 tablet_bootstrap.cc:492] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Bootstrap starting.
I20260929 02:53:07.843860 21019 log.cc:824] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:07.860574 21019 tablet_bootstrap.cc:492] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:53:07.861189 21019 tablet_bootstrap.cc:492] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Bootstrap complete.
I20260929 02:53:07.861618 21019 ts_tablet_manager.cc:1397] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.004s
I20260929 02:53:07.869982 21019 raft_consensus.cc:348] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:53:07.871143 21019 raft_consensus.cc:729] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0878e79b7944b62bc7cf74e221cad3b, State: Initialized, Role: FOLLOWER
I20260929 02:53:07.871974 21019 consensus_queue.cc:260] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } }
I20260929 02:53:07.872217 20977 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:41597
I20260929 02:53:07.872242 21126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:41597 every 8 connection(s)
I20260929 02:53:07.873498 20977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:53:07.873607 21019 ts_tablet_manager.cc:1428] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 02:53:07.874666 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 20977
I20260929 02:53:07.875430 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:43449
--local_ip_for_outbound_sockets=127.4.197.130
--tserver_master_addrs=127.4.197.190:34891
--webserver_port=42077
--webserver_interface=127.4.197.130
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:53:07 dist-test-slave-t6c1 krb5kdc[20353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:07.896492 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:07.876618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:50037 (local address 127.4.197.190:34891)
0929 02:53:07.878447 (+ 1829us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:07.878459 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:07.878496 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:53:07.878764 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:07.878776 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:07.879155 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:07.879489 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:07.881179 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:07.882449 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:07.884567 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:07.885235 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:07.885378 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:07.889205 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:07.889212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:07.889227 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:07.889270 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:07.892080 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:07.893049 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:07.893056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:07.893062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:07.893228 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:07.893674 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:07.893676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:07.893678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:07.894113 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:07.894263 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:07.895224 (+ 961us) server_negotiation.cc:299] Negotiation successful
0929 02:53:07.895554 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1648,"thread_start_us":1533,"threads_started":1}
I20260929 02:53:07.896857 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:07.876508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:50037)
0929 02:53:07.877335 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:07.877378 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:07.877851 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:07.879643 (+ 1792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:07.879654 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:07.880368 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:07.880959 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:07.880978 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:07.882616 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:07.882622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:07.884383 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:07.884398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:07.885446 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:07.885452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:07.885711 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:07.886423 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:53:07.886451 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:07.888944 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:07.892234 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:07.892244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:07.892261 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:07.892911 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:07.893331 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:07.893335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:07.893338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:07.893578 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:07.894242 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:07.894250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:07.894703 (+ 453us) client_negotiation.cc:769] Sending connection context
0929 02:53:07.895070 (+ 367us) client_negotiation.cc:240] Negotiation successful
0929 02:53:07.895617 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":347,"threads_started":1}
I20260929 02:53:07.899703 21128 heartbeater.cc:347] Connected to a master server at 127.4.197.190:34891
I20260929 02:53:07.899992 21128 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:07.900645 21128 heartbeater.cc:511] Master 127.4.197.190:34891 requested a full tablet report, sending...
I20260929 02:53:07.903576 20940 ts_manager.cc:194] Registered new tserver with Master: d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597)
I20260929 02:53:07.907799 20940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:50037
I20260929 02:53:07.919450 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:07.906912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.0.0.1:34760)
0929 02:53:07.912890 (+ 5978us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:07.912927 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:07.913095 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:07.913680 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:07.913685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:07.913708 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:07.913993 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:07.914005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:07.915774 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:07.915778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:07.917104 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:07.917116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:07.918754 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:07.918758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:07.918892 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:07.918966 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 02:53:07.919061 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:53:07.919190 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5746,"thread_start_us":5613,"threads_started":1}
I20260929 02:53:07.919586 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:07.907224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.4.197.129:41597)
0929 02:53:07.912410 (+ 5186us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:07.912419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:07.913033 (+ 614us) server_negotiation.cc:407] Connection header received
0929 02:53:07.913276 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:07.913284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:07.913494 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:07.913672 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:07.914148 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:07.915655 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:07.917251 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:07.918548 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:07.918673 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:07.918867 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:07.919121 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 02:53:07.919260 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5001,"thread_start_us":4751,"threads_started":1}
I20260929 02:53:07.920707 21062 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa5f7068ecf4a8095bf66d247b7ff92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:52:46 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:34760
I20260929 02:53:07.922575 21135 tablet_replica.cc:330] stopping tablet replica
I20260929 02:53:07.922946 21135 raft_consensus.cc:2229] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:53:07.923303 21135 raft_consensus.cc:2258] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:07.924451 21135 ts_tablet_manager.cc:1905] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:07.931412 21135 ts_tablet_manager.cc:1918] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:07.931558 21135 log.cc:1196] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/6fa5f7068ecf4a8095bf66d247b7ff92
I20260929 02:53:07.931951 21135 ts_tablet_manager.cc:1939] T 6fa5f7068ecf4a8095bf66d247b7ff92 P d0878e79b7944b62bc7cf74e221cad3b: Deleting consensus metadata
I20260929 02:53:07.934172 20925 catalog_manager.cc:4841] TS d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597): tablet 6fa5f7068ecf4a8095bf66d247b7ff92 (table test-table [id=e125aa5beb1a496c9c1401bd655461f2]) successfully deleted
W20260929 02:53:08.201122 21132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:08.201409 21132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:08.201503 21132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:08.208746 21132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:08.208849 21132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:08.208968 21132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:53:08.218441 21132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:43449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=42077
--tserver_master_addrs=127.4.197.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:08.219631 21132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:08.220868 21132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:08.233592 21141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:08.233630 21143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:08.233652 21140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:08.234292 21132 server_base.cc:1034] running on GCE node
I20260929 02:53:08.234946 21132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:08.235980 21132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:08.237210 21132 hybrid_clock.cc:648] HybridClock initialized: now 1790650388237158 us; error 49 us; skew 500 ppm
I20260929 02:53:08.237533 21132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:08 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:08.241139 21132 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:53:08.242725 21132 webserver.cc:460] Webserver started at http://127.4.197.130:42077/ using document root <none> and password file <none>
I20260929 02:53:08.243479 21132 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:08.243613 21132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:08.248976 21132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:53:08.251573 21150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:08.252815 21132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:53:08.252960 21132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "9ac614ef30c048558e64976e05863a4d"
format_stamp: "Formatted at 2026-09-29 02:52:45 on dist-test-slave-t6c1"
server_key: "d9869e03b78c2639426a5413b2197861"
server_key_iv: "dc8a3a4e60eb6a63853a78991e4129c7"
server_key_version: "encryptionkey@0"
I20260929 02:53:08.253885 21132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:08.286460 21132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:08.290469 21132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:08.290714 21132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:08.291738 21132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:08.292987 21132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:08.293084 21132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:08.293190 21132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:08.293257 21132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:08.322208 21132 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:43449
I20260929 02:53:08.322256 21262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:43449 every 8 connection(s)
I20260929 02:53:08.323781 21132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:53:08.329789 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 21132
I20260929 02:53:08.330487 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:39235
--local_ip_for_outbound_sockets=127.4.197.131
--tserver_master_addrs=127.4.197.190:34891
--webserver_port=42255
--webserver_interface=127.4.197.131
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:53:08 dist-test-slave-t6c1 krb5kdc[20353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:08.344441 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.326290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:46431 (local address 127.4.197.190:34891)
0929 02:53:08.326519 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:08.326525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:08.327909 (+ 1384us) server_negotiation.cc:407] Connection header received
0929 02:53:08.329840 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:08.329847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:08.329946 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:08.330061 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:08.331725 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.332695 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.334223 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.334835 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.334991 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.338630 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:08.338635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:08.338639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:08.338674 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:08.341212 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:08.342099 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:08.342120 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:08.342124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:08.342256 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:08.342710 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:08.342714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:08.342716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:08.343045 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:08.343171 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:08.343984 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 02:53:08.344162 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:53:08.345806 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.326908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:46431)
0929 02:53:08.327720 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:08.327761 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:08.329527 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:08.330277 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:08.330286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:08.330882 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:08.331530 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.331548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.332845 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.332848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.334085 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.334097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.335079 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.335085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.335298 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.335956 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:53:08.335978 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:08.338363 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:08.341368 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:08.341377 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:08.341395 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:08.341996 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:08.342373 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:08.342377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:08.342380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:08.342615 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:08.343158 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:08.343165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:08.343607 (+ 442us) client_negotiation.cc:769] Sending connection context
0929 02:53:08.343925 (+ 318us) client_negotiation.cc:240] Negotiation successful
0929 02:53:08.344453 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":292,"threads_started":1}
I20260929 02:53:08.347345 21263 heartbeater.cc:347] Connected to a master server at 127.4.197.190:34891
I20260929 02:53:08.347644 21263 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:08.348366 21263 heartbeater.cc:511] Master 127.4.197.190:34891 requested a full tablet report, sending...
I20260929 02:53:08.349829 20940 ts_manager.cc:194] Registered new tserver with Master: 9ac614ef30c048558e64976e05863a4d (127.4.197.130:43449)
I20260929 02:53:08.350674 20940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:46431
W20260929 02:53:08.655997 21266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:08.656312 21266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:08.656441 21266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:08.663937 21266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:08.664064 21266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:08.664218 21266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:53:08.674014 21266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:39235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=42255
--tserver_master_addrs=127.4.197.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:08.675424 21266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:08.676762 21266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:08.689983 21271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:08.689935 21272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:08.690032 21274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:08.690320 21266 server_base.cc:1034] running on GCE node
I20260929 02:53:08.691184 21266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:08.692050 21266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:08.693230 21266 hybrid_clock.cc:648] HybridClock initialized: now 1790650388693168 us; error 50 us; skew 500 ppm
I20260929 02:53:08.693517 21266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:08 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:08.696501 21266 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:53:08.697876 21266 webserver.cc:460] Webserver started at http://127.4.197.131:42255/ using document root <none> and password file <none>
I20260929 02:53:08.698521 21266 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:08.698633 21266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:08.703750 21266 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:53:08.706326 21281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:08.707612 21266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:53:08.707739 21266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "4f8a55e7c21746629a6757313665b2ae"
format_stamp: "Formatted at 2026-09-29 02:52:45 on dist-test-slave-t6c1"
server_key: "7304392a848a5b466b20ed1df81b4aa0"
server_key_iv: "1444f133703bf2f580acfbf311c76753"
server_key_version: "encryptionkey@0"
I20260929 02:53:08.708649 21266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:08.720860 21266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:08.724717 21266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:08.724932 21266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:08.725768 21266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:08.727056 21266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:08.727140 21266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:08.727236 21266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:08.727309 21266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:08.756716 21266 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:39235
I20260929 02:53:08.756827 21393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:39235 every 8 connection(s)
I20260929 02:53:08.758196 21266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:53:08.765592 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 21266
Sep 29 02:53:08 dist-test-slave-t6c1 krb5kdc[20353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:08.779403 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.760655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:42669 (local address 127.4.197.190:34891)
0929 02:53:08.760834 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:08.760839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:08.761948 (+ 1109us) server_negotiation.cc:407] Connection header received
0929 02:53:08.763776 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:08.763782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:08.763863 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:08.763949 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:08.765462 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.766239 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.768278 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.768668 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.768781 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.772827 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:08.772833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:08.772838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:08.772899 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:08.776339 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:08.777069 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:08.777075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:08.777079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:08.777220 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:08.777636 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:08.777640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:08.777642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:08.777915 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:08.778055 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:08.778799 (+ 744us) server_negotiation.cc:299] Negotiation successful
0929 02:53:08.779055 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59,"spinlock_wait_cycles":1792}
I20260929 02:53:08.780412 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.761129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:42669)
0929 02:53:08.761793 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:08.761831 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:08.763514 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:08.764189 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:08.764198 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:08.764687 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:08.765283 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.765303 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.766395 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.766399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.768142 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.768155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.768890 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.768897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.769127 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.769809 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:53:08.769841 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:08.772582 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:08.776470 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:08.776481 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:08.776497 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:08.776945 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:08.777322 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:08.777327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:08.777330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:08.777519 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:08.778052 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:08.778060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:08.778408 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 02:53:08.778669 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 02:53:08.779290 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":252,"threads_started":1}
I20260929 02:53:08.781700 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.766657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.4.197.190:34891)
0929 02:53:08.767266 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:08.767272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:08.767541 (+ 269us) server_negotiation.cc:407] Connection header received
0929 02:53:08.767692 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:08.767698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:08.767771 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:08.767882 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:08.769385 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.770417 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.772116 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.772743 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.772841 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.776331 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:08.776338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:08.776343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:08.776384 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:08.778892 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:08.779725 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:08.779728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:08.779731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:08.779860 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:08.780221 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:08.780224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:08.780226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:08.780477 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:08.780708 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:08.781119 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 02:53:08.781348 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"spinlock_wait_cycles":1792,"thread_start_us":262,"threads_started":1}
I20260929 02:53:08.782020 21394 heartbeater.cc:347] Connected to a master server at 127.4.197.190:34891
I20260929 02:53:08.782313 21394 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:08.783070 21394 heartbeater.cc:511] Master 127.4.197.190:34891 requested a full tablet report, sending...
I20260929 02:53:08.784376 20940 ts_manager.cc:194] Registered new tserver with Master: 4f8a55e7c21746629a6757313665b2ae (127.4.197.131:39235)
I20260929 02:53:08.785138 20940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:42669
I20260929 02:53:08.796286 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:53:08.802475 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.797209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.4.197.190:34891)
0929 02:53:08.797417 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:08.797425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:08.797846 (+ 421us) server_negotiation.cc:407] Connection header received
0929 02:53:08.798016 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:08.798019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:08.798163 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:08.798306 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:53:08.798636 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.799394 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.800838 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.801236 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.801338 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.801725 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:53:08.801926 (+ 201us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 02:53:08.802184 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59964: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 02:53:08.802613 21397 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59964: token expired
W20260929 02:53:08.803105 20858 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.197.190:34891)
I20260929 02:53:08.815330 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.803532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.4.197.190:34891)
0929 02:53:08.803701 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:08.803705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:08.803734 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:53:08.803874 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:08.803877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:08.803931 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:08.804002 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:08.805452 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.806253 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.807622 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.807997 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.808088 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.810498 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:08.810502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:08.810506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:08.810537 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:08.813101 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:08.813708 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:08.813712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:08.813714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:08.813836 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:08.814089 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:08.814091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:08.814093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:08.814374 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:08.814552 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:08.814795 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 02:53:08.815033 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:53:08.818320 20940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:53:08.825469 20940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:53:08.838117 21062 tablet_service.cc:1467] Processing CreateTablet for tablet 6a28a0f817394814be5b1e8ac83886f4 (DEFAULT_TABLE table=test-table [id=02a5654e260646e2be2fd782eeebaab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:08.839111 21062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a28a0f817394814be5b1e8ac83886f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:08.848978 21408 tablet_bootstrap.cc:492] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: Bootstrap starting.
I20260929 02:53:08.851853 21408 tablet_bootstrap.cc:654] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:08.856230 21408 tablet_bootstrap.cc:492] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: No bootstrap required, opened a new log
I20260929 02:53:08.856477 21408 ts_tablet_manager.cc:1397] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260929 02:53:08.857483 21408 raft_consensus.cc:348] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:08.857755 21408 raft_consensus.cc:374] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:08.857890 21408 raft_consensus.cc:729] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0878e79b7944b62bc7cf74e221cad3b, State: Initialized, Role: FOLLOWER
I20260929 02:53:08.858180 21408 consensus_queue.cc:260] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:08.859036 21128 heartbeater.cc:503] Master 127.4.197.190:34891 was elected leader, sending a full tablet report...
I20260929 02:53:08.859647 21408 ts_tablet_manager.cc:1428] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:53:08.861943 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.836400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:39235 (local address 127.0.0.1:54640)
0929 02:53:08.852479 (+ 16079us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:08.852502 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:08.852654 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:08.854853 (+ 2199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:08.854857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:08.854885 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:08.855227 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.855241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.857170 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.857175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.859132 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.859147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.860873 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.860880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.861297 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.861333 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:53:08.861412 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:53:08.861548 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15930,"mutex_wait_us":19,"thread_start_us":19304,"threads_started":1}
I20260929 02:53:08.861995 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.836625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.4.197.131:39235)
0929 02:53:08.854162 (+ 17537us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:08.854172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:08.854209 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:53:08.854322 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:08.854331 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:08.854626 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:08.854846 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:08.855481 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.857044 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.859310 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.860667 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.860813 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.861055 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:08.861514 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 02:53:08.861687 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17342,"thread_start_us":17194,"threads_started":1}
I20260929 02:53:08.863592 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.837015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:43449 (local address 127.0.0.1:55612)
0929 02:53:08.855974 (+ 18959us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:08.855995 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:08.856188 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:08.856969 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:08.856973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:08.856997 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:08.857415 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.857427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.859235 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.859241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.861073 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.861084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.862773 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.862780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.863005 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.863035 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:53:08.863124 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 02:53:08.863230 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18843,"mutex_wait_us":15,"thread_start_us":15163,"threads_started":1}
I20260929 02:53:08.864389 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.837757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.4.197.130:43449)
0929 02:53:08.855524 (+ 17767us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:08.855534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:08.856088 (+ 554us) server_negotiation.cc:407] Connection header received
0929 02:53:08.856383 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:08.856391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:08.856651 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:08.856835 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:08.857595 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.859072 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.861203 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.862564 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.862681 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.862866 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:08.863955 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0929 02:53:08.864121 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17632,"thread_start_us":17543,"threads_started":1}
I20260929 02:53:08.866132 21329 tablet_service.cc:1467] Processing CreateTablet for tablet 6a28a0f817394814be5b1e8ac83886f4 (DEFAULT_TABLE table=test-table [id=02a5654e260646e2be2fd782eeebaab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:08.867537 21329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a28a0f817394814be5b1e8ac83886f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:08.867703 21198 tablet_service.cc:1467] Processing CreateTablet for tablet 6a28a0f817394814be5b1e8ac83886f4 (DEFAULT_TABLE table=test-table [id=02a5654e260646e2be2fd782eeebaab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:08.869102 21198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a28a0f817394814be5b1e8ac83886f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 02:53:08.876543 21129 tablet.cc:2369] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:53:08.880936 21410 raft_consensus.cc:482] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:53:08.881232 21410 raft_consensus.cc:504] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:08.885826 21412 tablet_bootstrap.cc:492] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: Bootstrap starting.
I20260929 02:53:08.886502 21410 leader_election.cc:288] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8a55e7c21746629a6757313665b2ae (127.4.197.131:39235), 9ac614ef30c048558e64976e05863a4d (127.4.197.130:43449)
I20260929 02:53:08.891059 21412 tablet_bootstrap.cc:654] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:08.891767 21411 tablet_bootstrap.cc:492] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: Bootstrap starting.
I20260929 02:53:08.893486 21412 log.cc:824] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:08.895161 21411 tablet_bootstrap.cc:654] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:08.896548 21411 log.cc:824] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:08.902324 21412 tablet_bootstrap.cc:492] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: No bootstrap required, opened a new log
I20260929 02:53:08.902601 21412 ts_tablet_manager.cc:1397] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.009s
I20260929 02:53:08.905956 21411 tablet_bootstrap.cc:492] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: No bootstrap required, opened a new log
I20260929 02:53:08.906273 21411 ts_tablet_manager.cc:1397] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260929 02:53:08.908686 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.884264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:41055 (local address 127.4.197.131:39235)
0929 02:53:08.884463 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:08.884469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:08.899116 (+ 14647us) server_negotiation.cc:407] Connection header received
0929 02:53:08.900023 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:08.900028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:08.900105 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:08.900162 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:08.901579 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.902747 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.906421 (+ 3674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.907579 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.907716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.907793 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:08.908305 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 02:53:08.908437 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:53:08.909109 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.884226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:48319 (local address 127.4.197.130:43449)
0929 02:53:08.884422 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:08.884428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:08.899878 (+ 15450us) server_negotiation.cc:407] Connection header received
0929 02:53:08.899954 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:08.899959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:08.900033 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:08.900120 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:08.902977 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.903970 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.906768 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.908016 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.908138 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.908213 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:08.908715 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 02:53:08.908860 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:53:08.909739 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.883619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:39235 (local address 127.4.197.129:41055)
0929 02:53:08.898990 (+ 15371us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:08.899014 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:08.899860 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:08.900357 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:08.900362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:08.900388 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:08.900786 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.900799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.904119 (+ 3320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.904122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.906231 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.906246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.907949 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.907955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.908122 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.908153 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:53:08.909060 (+ 907us) client_negotiation.cc:240] Negotiation successful
0929 02:53:08.909324 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15195,"mutex_wait_us":30,"thread_start_us":15647,"threads_started":1}
I20260929 02:53:08.909847 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:08.883655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:43449 (local address 127.4.197.129:48319)
0929 02:53:08.899571 (+ 15916us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:08.899588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:08.899726 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:08.900963 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:08.900968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:08.900991 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:08.901395 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.901404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.904342 (+ 2938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.904347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.906567 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:08.906582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:08.908379 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:08.908385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:08.908545 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:08.908575 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:53:08.909320 (+ 745us) client_negotiation.cc:240] Negotiation successful
0929 02:53:08.909451 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15782,"mutex_wait_us":56,"thread_start_us":15010,"threads_started":1}
I20260929 02:53:08.910903 21349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a28a0f817394814be5b1e8ac83886f4" candidate_uuid: "d0878e79b7944b62bc7cf74e221cad3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8a55e7c21746629a6757313665b2ae" is_pre_election: true
W20260929 02:53:08.912242 21013 leader_election.cc:341] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4f8a55e7c21746629a6757313665b2ae (127.4.197.131:39235): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:53:08.910887 21218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a28a0f817394814be5b1e8ac83886f4" candidate_uuid: "d0878e79b7944b62bc7cf74e221cad3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac614ef30c048558e64976e05863a4d" is_pre_election: true
W20260929 02:53:08.913666 21016 leader_election.cc:341] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ac614ef30c048558e64976e05863a4d (127.4.197.130:43449): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:53:08.913800 21016 leader_election.cc:302] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0878e79b7944b62bc7cf74e221cad3b; no voters: 4f8a55e7c21746629a6757313665b2ae, 9ac614ef30c048558e64976e05863a4d
I20260929 02:53:08.913710 21412 raft_consensus.cc:348] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:08.914072 21412 raft_consensus.cc:374] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:08.914158 21412 raft_consensus.cc:729] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8a55e7c21746629a6757313665b2ae, State: Initialized, Role: FOLLOWER
I20260929 02:53:08.914431 21410 raft_consensus.cc:2726] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:53:08.914718 21412 consensus_queue.cc:260] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:08.915721 21394 heartbeater.cc:503] Master 127.4.197.190:34891 was elected leader, sending a full tablet report...
I20260929 02:53:08.916254 21412 ts_tablet_manager.cc:1428] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20260929 02:53:08.917223 21411 raft_consensus.cc:348] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:08.917508 21411 raft_consensus.cc:374] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:08.917589 21411 raft_consensus.cc:729] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac614ef30c048558e64976e05863a4d, State: Initialized, Role: FOLLOWER
I20260929 02:53:08.918102 21411 consensus_queue.cc:260] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:08.919100 21263 heartbeater.cc:503] Master 127.4.197.190:34891 was elected leader, sending a full tablet report...
I20260929 02:53:08.919551 21411 ts_tablet_manager.cc:1428] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260929 02:53:09.009768 21395 tablet.cc:2369] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:53:09.075747 21264 tablet.cc:2369] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:53:09.356511 21418 raft_consensus.cc:482] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:53:09.356859 21418 raft_consensus.cc:504] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:09.358464 21418 leader_election.cc:288] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8a55e7c21746629a6757313665b2ae (127.4.197.131:39235), d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597)
I20260929 02:53:09.366398 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.358786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:33885 (local address 127.4.197.129:41597)
0929 02:53:09.359333 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:09.359339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:09.359898 (+ 559us) server_negotiation.cc:407] Connection header received
0929 02:53:09.360323 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:09.360328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:09.360398 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:09.360471 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:09.361027 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.362057 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.364055 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.365146 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.365280 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.365365 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:09.365952 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 02:53:09.366092 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":365,"threads_started":1}
I20260929 02:53:09.366616 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.358984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:46031 (local address 127.4.197.131:39235)
0929 02:53:09.359143 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:09.359150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:09.359898 (+ 748us) server_negotiation.cc:407] Connection header received
0929 02:53:09.360096 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:09.360100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:09.360165 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:09.360251 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:09.360797 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.361920 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.364502 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.365632 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.365784 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.365849 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:09.366215 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 02:53:09.366360 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 02:53:09.366681 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.358673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.4.197.130:33885)
0929 02:53:09.359765 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:09.359784 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:09.360167 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:09.360518 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:09.360522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:09.360542 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:09.360887 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:09.360895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.362182 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.362190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:09.363868 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:09.363885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.365553 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.365559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:09.365739 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.365774 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:53:09.365910 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 02:53:09.366049 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":2,"thread_start_us":678,"threads_started":1}
I20260929 02:53:09.366865 21417 raft_consensus.cc:482] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:53:09.367146 21417 raft_consensus.cc:504] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:09.367588 21082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a28a0f817394814be5b1e8ac83886f4" candidate_uuid: "9ac614ef30c048558e64976e05863a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0878e79b7944b62bc7cf74e221cad3b" is_pre_election: true
I20260929 02:53:09.367961 21082 raft_consensus.cc:2454] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ac614ef30c048558e64976e05863a4d in term 0.
I20260929 02:53:09.368506 21152 leader_election.cc:302] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac614ef30c048558e64976e05863a4d, d0878e79b7944b62bc7cf74e221cad3b; no voters:
I20260929 02:53:09.368826 21417 leader_election.cc:288] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac614ef30c048558e64976e05863a4d (127.4.197.130:43449), d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597)
I20260929 02:53:09.368914 21418 raft_consensus.cc:2781] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:53:09.369058 21418 raft_consensus.cc:482] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:53:09.369163 21418 raft_consensus.cc:3037] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:09.372509 21418 raft_consensus.cc:504] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:09.373595 21421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.358673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:39235 (local address 127.4.197.130:46031)
0929 02:53:09.359803 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:09.359818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:09.359969 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:09.360278 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:09.360281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:09.360299 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:09.360617 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:09.360633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.362052 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.362058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:09.364336 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:09.364351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.365850 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.365856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:09.366024 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.366056 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:53:09.368978 (+ 2922us) client_negotiation.cc:240] Negotiation successful
0929 02:53:09.369111 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":162,"thread_start_us":655,"threads_started":1}
I20260929 02:53:09.374146 21082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a28a0f817394814be5b1e8ac83886f4" candidate_uuid: "9ac614ef30c048558e64976e05863a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0878e79b7944b62bc7cf74e221cad3b"
I20260929 02:53:09.374365 21082 raft_consensus.cc:3037] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:09.376865 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.369284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:39313 (local address 127.4.197.129:41597)
0929 02:53:09.370147 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:09.370152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:09.370362 (+ 210us) server_negotiation.cc:407] Connection header received
0929 02:53:09.370419 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:09.370424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:09.370487 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:09.370555 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:09.371242 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.372203 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.374590 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.375818 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.375950 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.376021 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:09.376498 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 02:53:09.376624 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757}
I20260929 02:53:09.377187 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.369447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:48307 (local address 127.4.197.130:43449)
0929 02:53:09.369600 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:09.369605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:09.369665 (+ 60us) server_negotiation.cc:407] Connection header received
0929 02:53:09.369920 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:09.369929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:09.370002 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:09.370066 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:09.370715 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.371921 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.374887 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.376073 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.376191 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.376255 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:09.376822 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 02:53:09.376944 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:53:09.377835 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.369031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:41597 (local address 127.4.197.131:39313)
0929 02:53:09.370133 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:09.370150 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:09.370267 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:09.370699 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:09.370705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:09.370727 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:09.371081 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:09.371092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.372378 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.372385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:09.374430 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:09.374444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.376148 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.376153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:09.376313 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.376342 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:53:09.377026 (+ 684us) client_negotiation.cc:240] Negotiation successful
0929 02:53:09.377159 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":309,"thread_start_us":524,"threads_started":1}
I20260929 02:53:09.378140 21082 raft_consensus.cc:2454] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac614ef30c048558e64976e05863a4d in term 1.
I20260929 02:53:09.378377 21418 leader_election.cc:288] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [CANDIDATE]: Term 1 election: Requested vote from peers 4f8a55e7c21746629a6757313665b2ae (127.4.197.131:39235), d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597)
I20260929 02:53:09.378365 21348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a28a0f817394814be5b1e8ac83886f4" candidate_uuid: "9ac614ef30c048558e64976e05863a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8a55e7c21746629a6757313665b2ae"
I20260929 02:53:09.378559 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.369025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:43449 (local address 127.4.197.131:48307)
0929 02:53:09.369563 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:09.369584 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:09.369772 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:09.370230 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:09.370236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:09.370290 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:09.370551 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:09.370561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.372086 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.372091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:09.374734 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:09.374748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.376472 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.376479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:09.376659 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.376691 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:53:09.377320 (+ 629us) client_negotiation.cc:240] Negotiation successful
0929 02:53:09.377459 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":521,"spinlock_wait_cycles":26368,"thread_start_us":574,"threads_started":1}
I20260929 02:53:09.378648 21348 raft_consensus.cc:3037] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:09.378382 21349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a28a0f817394814be5b1e8ac83886f4" candidate_uuid: "9ac614ef30c048558e64976e05863a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8a55e7c21746629a6757313665b2ae" is_pre_election: true
I20260929 02:53:09.379222 21152 leader_election.cc:302] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac614ef30c048558e64976e05863a4d, d0878e79b7944b62bc7cf74e221cad3b; no voters:
I20260929 02:53:09.379557 21218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a28a0f817394814be5b1e8ac83886f4" candidate_uuid: "4f8a55e7c21746629a6757313665b2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac614ef30c048558e64976e05863a4d" is_pre_election: true
I20260929 02:53:09.379837 21218 raft_consensus.cc:2379] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4f8a55e7c21746629a6757313665b2ae in current term 1: Already voted for candidate 9ac614ef30c048558e64976e05863a4d in this term.
I20260929 02:53:09.380400 21418 raft_consensus.cc:2781] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:53:09.380771 21082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a28a0f817394814be5b1e8ac83886f4" candidate_uuid: "4f8a55e7c21746629a6757313665b2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0878e79b7944b62bc7cf74e221cad3b" is_pre_election: true
I20260929 02:53:09.381054 21082 raft_consensus.cc:2379] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4f8a55e7c21746629a6757313665b2ae in current term 1: Already voted for candidate 9ac614ef30c048558e64976e05863a4d in this term.
I20260929 02:53:09.381418 21418 raft_consensus.cc:686] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 1 LEADER]: Becoming Leader. State: Replica: 9ac614ef30c048558e64976e05863a4d, State: Running, Role: LEADER
I20260929 02:53:09.381672 21283 leader_election.cc:302] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f8a55e7c21746629a6757313665b2ae; no voters: 9ac614ef30c048558e64976e05863a4d, d0878e79b7944b62bc7cf74e221cad3b
I20260929 02:53:09.381819 21348 raft_consensus.cc:2454] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac614ef30c048558e64976e05863a4d in term 1.
I20260929 02:53:09.381934 21418 consensus_queue.cc:237] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } }
I20260929 02:53:09.382357 21417 raft_consensus.cc:2726] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:53:09.386325 20938 catalog_manager.cc:5495] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ac614ef30c048558e64976e05863a4d (127.4.197.130). New cstate: current_term: 1 leader_uuid: "9ac614ef30c048558e64976e05863a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ac614ef30c048558e64976e05863a4d" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 } health_report { overall_health: UNKNOWN } } }
I20260929 02:53:09.432035 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.426581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.4.197.130:43449)
0929 02:53:09.426812 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:09.426818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:09.427114 (+ 296us) server_negotiation.cc:407] Connection header received
0929 02:53:09.427292 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:09.427295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:09.427396 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:09.427519 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:53:09.427833 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.428513 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.429859 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.430278 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.430387 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.430883 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:53:09.431447 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:53:09.431580 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:09.431679 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 02:53:09.431760 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 02:53:09.439898 21082 raft_consensus.cc:1261] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Refusing update from remote peer 9ac614ef30c048558e64976e05863a4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:53:09.439903 21348 raft_consensus.cc:1261] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 1 FOLLOWER]: Refusing update from remote peer 9ac614ef30c048558e64976e05863a4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:53:09.440640 21424 consensus_queue.cc:1035] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0878e79b7944b62bc7cf74e221cad3b" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:53:09.441009 21418 consensus_queue.cc:1035] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8a55e7c21746629a6757313665b2ae" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:53:09.469466 20940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:53:09.469769 20940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:53:09.473346 20940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Sending DeleteTablet for 3 replicas of tablet 6a28a0f817394814be5b1e8ac83886f4
I20260929 02:53:09.474385 21329 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a28a0f817394814be5b1e8ac83886f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:09 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:54640
I20260929 02:53:09.474429 21062 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a28a0f817394814be5b1e8ac83886f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:09 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:34760
I20260929 02:53:09.474630 21198 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a28a0f817394814be5b1e8ac83886f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:09 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:55612
I20260929 02:53:09.476068 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 20977
I20260929 02:53:09.487521 21443 tablet_replica.cc:330] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: stopping tablet replica
I20260929 02:53:09.488129 21443 raft_consensus.cc:2229] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:53:09.488775 21443 raft_consensus.cc:2258] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:09.492625 21443 ts_tablet_manager.cc:1905] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:09.499991 21443 ts_tablet_manager.cc:1918] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:09.500278 21443 log.cc:1196] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/6a28a0f817394814be5b1e8ac83886f4
I20260929 02:53:09.501327 21443 ts_tablet_manager.cc:1939] T 6a28a0f817394814be5b1e8ac83886f4 P 4f8a55e7c21746629a6757313665b2ae: Deleting consensus metadata
I20260929 02:53:09.503327 20924 catalog_manager.cc:4841] TS 4f8a55e7c21746629a6757313665b2ae (127.4.197.131:39235): tablet 6a28a0f817394814be5b1e8ac83886f4 (table test-table [id=02a5654e260646e2be2fd782eeebaab4]) successfully deleted
I20260929 02:53:09.503854 21444 tablet_replica.cc:330] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: stopping tablet replica
I20260929 02:53:09.504388 21444 raft_consensus.cc:2229] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:53:09.505134 21444 raft_consensus.cc:2258] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:09.510263 21444 ts_tablet_manager.cc:1905] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:09.512750 21442 tablet_replica.cc:330] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: stopping tablet replica
I20260929 02:53:09.512938 21442 raft_consensus.cc:2229] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:53:09.513298 21442 raft_consensus.cc:2258] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:09.514298 21442 ts_tablet_manager.cc:1905] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:09.517680 21444 ts_tablet_manager.cc:1918] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:09.517868 21444 log.cc:1196] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/6a28a0f817394814be5b1e8ac83886f4
I20260929 02:53:09.518469 21444 ts_tablet_manager.cc:1939] T 6a28a0f817394814be5b1e8ac83886f4 P 9ac614ef30c048558e64976e05863a4d: Deleting consensus metadata
I20260929 02:53:09.520233 20927 catalog_manager.cc:4841] TS 9ac614ef30c048558e64976e05863a4d (127.4.197.130:43449): tablet 6a28a0f817394814be5b1e8ac83886f4 (table test-table [id=02a5654e260646e2be2fd782eeebaab4]) successfully deleted
I20260929 02:53:09.521478 21442 ts_tablet_manager.cc:1918] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:09.521605 21442 log.cc:1196] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/6a28a0f817394814be5b1e8ac83886f4
I20260929 02:53:09.521678 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.477326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.4.197.129:41597)
0929 02:53:09.477511 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:09.477517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:09.499897 (+ 22380us) server_negotiation.cc:407] Connection header received
0929 02:53:09.500525 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:09.500532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:09.500659 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:09.500739 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:09.505680 (+ 4941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.506961 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.508824 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.509531 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.509649 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.514324 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:09.514342 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:09.514359 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:09.514400 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:09.518103 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:09.518983 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:09.518992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:09.518996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:09.519168 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:09.519632 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:09.519641 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:09.519644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:09.520103 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:09.520390 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:09.521012 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 02:53:09.521261 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:53:09.522024 21442 ts_tablet_manager.cc:1939] T 6a28a0f817394814be5b1e8ac83886f4 P d0878e79b7944b62bc7cf74e221cad3b: Deleting consensus metadata
I20260929 02:53:09.522329 21122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:09.636960 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 20977
I20260929 02:53:09.636914 20925 catalog_manager.cc:4841] TS d0878e79b7944b62bc7cf74e221cad3b (127.4.197.129:41597): tablet 6a28a0f817394814be5b1e8ac83886f4 (table test-table [id=02a5654e260646e2be2fd782eeebaab4]) successfully deleted
I20260929 02:53:09.658252 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 21132
I20260929 02:53:09.673697 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.659106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.4.197.130:43449)
0929 02:53:09.659265 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:09.659269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:09.659286 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:53:09.659422 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:09.659426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:09.659493 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:09.659579 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:09.661263 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.661990 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.663939 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.664438 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.664537 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.667867 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:09.667883 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:09.667898 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:09.667933 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:09.670894 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:09.671763 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:09.671770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:09.671772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:09.671911 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:09.672280 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:09.672283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:09.672285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:09.672652 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:09.672896 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:09.673255 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 02:53:09.673425 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 02:53:09.674499 21258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:09.788000 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 21132
I20260929 02:53:09.803196 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 21266
I20260929 02:53:09.815754 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:09.804058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.4.197.131:39235)
0929 02:53:09.804221 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:09.804225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:09.804243 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:53:09.804345 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:09.804349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:09.804411 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:09.804491 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:09.805701 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.806417 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.807698 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:09.808121 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:09.808215 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:09.810608 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:09.810622 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:09.810636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:09.810670 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:09.813434 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:09.814047 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:09.814053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:09.814055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:09.814184 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:09.814527 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:09.814531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:09.814533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:09.814863 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:09.815129 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:09.815321 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 02:53:09.815465 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:53:09.816355 21389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:09.905179 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 21266
I20260929 02:53:09.917768 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 20907
I20260929 02:53:09.918851 20970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:10.029270 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 20907
I20260929 02:53:30.046859 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34891
--webserver_interface=127.4.197.190
--webserver_port=37573
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:53:30.374533 21450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:30.374878 21450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:30.375021 21450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:30.382457 21450 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:53:30.382562 21450 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:53:30.382598 21450 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:53:30.382644 21450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:53:30.382699 21450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:30.382763 21450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:30.382808 21450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:53:30.382858 21450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:53:30.392958 21450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:34891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=37573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:30.394170 21450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:30.395471 21450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:30.406970 21455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:30.406836 21456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:30.406877 21458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:30.407572 21450 server_base.cc:1034] running on GCE node
I20260929 02:53:30.408277 21450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:30.409612 21450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:30.410820 21450 hybrid_clock.cc:648] HybridClock initialized: now 1790650410410750 us; error 47 us; skew 500 ppm
I20260929 02:53:30.411137 21450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:30 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650410, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:30.419628 21450 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:53:30.421012 21450 webserver.cc:460] Webserver started at http://127.4.197.190:37573/ using document root <none> and password file <none>
I20260929 02:53:30.421648 21450 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:30.421747 21450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:30.427320 21450 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:53:30.430411 21465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:30.431840 21450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:53:30.432040 21450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "b9f964978811488d919e5227bddf86a3"
format_stamp: "Formatted at 2026-09-29 02:52:44 on dist-test-slave-t6c1"
server_key: "2d4ae9da6dcec9fc1da3ab0102ba3435"
server_key_iv: "aae4c52ac425b9154ed72fe5861d843e"
server_key_version: "encryptionkey@0"
I20260929 02:53:30.433060 21450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:30.444832 21450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:30.448978 21450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:30.449242 21450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:30.466868 21450 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:34891
I20260929 02:53:30.466899 21516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:34891 every 8 connection(s)
I20260929 02:53:30.468369 21450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:53:30.471377 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 21450
I20260929 02:53:30.472520 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:41597
--local_ip_for_outbound_sockets=127.4.197.129
--tserver_master_addrs=127.4.197.190:34891
--webserver_port=38393
--webserver_interface=127.4.197.129
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:53:30.477568 21517 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:53:30.482537 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Bootstrap starting.
I20260929 02:53:30.504112 21517 log.cc:824] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:30.520829 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:53:30.521396 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Bootstrap complete.
I20260929 02:53:30.530858 21517 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:53:30.531972 21517 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f964978811488d919e5227bddf86a3, State: Initialized, Role: FOLLOWER
I20260929 02:53:30.532689 21517 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:53:30.532932 21517 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:53:30.533066 21517 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:53:30.533238 21517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:53:30.537415 21517 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:53:30.537957 21517 leader_election.cc:302] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9f964978811488d919e5227bddf86a3; no voters:
I20260929 02:53:30.538677 21517 leader_election.cc:288] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 02:53:30.538847 21521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:53:30.541000 21517 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:53:30.543218 21521 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [term 3 LEADER]: Becoming Leader. State: Replica: b9f964978811488d919e5227bddf86a3, State: Running, Role: LEADER
I20260929 02:53:30.543985 21521 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } }
I20260929 02:53:30.546799 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9f964978811488d919e5227bddf86a3. Latest consensus state: current_term: 3 leader_uuid: "b9f964978811488d919e5227bddf86a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } } }
I20260929 02:53:30.547263 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:30.547851 21522 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b9f964978811488d919e5227bddf86a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f964978811488d919e5227bddf86a3" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34891 } } }
I20260929 02:53:30.548163 21522 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:30.555889 21532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:53:30.564561 21532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:53:30.566075 21532 catalog_manager.cc:1259] Loaded cluster ID: 732e954e031440a382e66ff71caadd6c
I20260929 02:53:30.566255 21532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:53:30.570447 21532 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:53:30.571305 21539 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 02:53:30.572537 21532 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Loaded TSK: 0
I20260929 02:53:30.572676 21532 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b9f964978811488d919e5227bddf86a3: Loaded TSK: 1
F20260929 02:53:30.573807 21532 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f440c9029ad google::LogMessage::Fail() at ??:0
@ 0x7f440c90588a google::LogMessage::SendToLog() at ??:0
@ 0x7f440c9024de google::LogMessage::Flush() at ??:0
@ 0x7f440c903a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f44106667d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f441065573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f44223311c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4422330783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f44223bc2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f44223bc0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f44224065e9 std::function<>::operator()() at ??:0
@ 0x7f4422334cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4422332c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f44223ba610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f44223ba439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f440f54e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f440f55191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f440f551799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f440f52ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f44128066db start_thread at ??:0
@ 0x7f440b79c61f clone at ??:0
W20260929 02:53:30.834240 21519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:30.834571 21519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:30.834697 21519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:30.842211 21519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:30.842366 21519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:30.842532 21519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:53:30.852865 21519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:41597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=38393
--tserver_master_addrs=127.4.197.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:30.854158 21519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:30.855491 21519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:30.868078 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:30.868124 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:30.868350 21568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:30.869040 21519 server_base.cc:1034] running on GCE node
I20260929 02:53:30.869620 21519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:30.870508 21519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:30.871717 21519 hybrid_clock.cc:648] HybridClock initialized: now 1790650410871666 us; error 34 us; skew 500 ppm
I20260929 02:53:30.872020 21519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:30 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650410, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:30.874981 21519 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:53:30.876470 21519 webserver.cc:460] Webserver started at http://127.4.197.129:38393/ using document root <none> and password file <none>
I20260929 02:53:30.877133 21519 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:30.877254 21519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:30.882639 21519 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:53:30.885583 21575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:30.887105 21519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:53:30.887282 21519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "d0878e79b7944b62bc7cf74e221cad3b"
format_stamp: "Formatted at 2026-09-29 02:52:44 on dist-test-slave-t6c1"
server_key: "362fd47f7f2e8afe8fdff3688e3a6870"
server_key_iv: "176507111dddaa0ab1cdd9d16b14f270"
server_key_version: "encryptionkey@0"
I20260929 02:53:30.888383 21519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:30.914831 21519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:30.919066 21519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:30.919330 21519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:30.920351 21519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:30.921813 21519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:30.921911 21519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:30.922019 21519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:30.922103 21519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:30.953635 21519 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:41597
I20260929 02:53:30.953693 21687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:41597 every 8 connection(s)
I20260929 02:53:30.955155 21519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:53:30.959254 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 21519
I20260929 02:53:30.959969 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:43449
--local_ip_for_outbound_sockets=127.4.197.130
--tserver_master_addrs=127.4.197.190:34891
--webserver_port=42077
--webserver_interface=127.4.197.130
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:53:30.960398 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:30.958264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:36285)
0929 02:53:30.959069 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:30.959352 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":305,"threads_started":1}
W20260929 02:53:30.961073 21688 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:34891 (0 consecutive failures): Network error: Failed to ping master at 127.4.197.190:34891: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
I20260929 02:53:30.961943 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:30.961599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:43823)
0929 02:53:30.961753 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:30.961824 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 02:53:30.962751 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:30.962444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:50027)
0929 02:53:30.962573 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:30.962647 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 02:53:30.963016 21688 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:53:31.298748 21691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:31.299036 21691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:31.299137 21691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:31.306342 21691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:31.306474 21691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:31.306603 21691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:53:31.316560 21691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:43449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=42077
--tserver_master_addrs=127.4.197.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:31.317983 21691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:31.319470 21691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:31.333213 21696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:31.333266 21697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:31.333245 21699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:31.334270 21691 server_base.cc:1034] running on GCE node
I20260929 02:53:31.334959 21691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:31.335973 21691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:31.337178 21691 hybrid_clock.cc:648] HybridClock initialized: now 1790650411337111 us; error 60 us; skew 500 ppm
I20260929 02:53:31.337497 21691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:31 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650411, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:31.341459 21691 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:53:31.342898 21691 webserver.cc:460] Webserver started at http://127.4.197.130:42077/ using document root <none> and password file <none>
I20260929 02:53:31.343631 21691 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:31.343729 21691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:31.348905 21691 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 02:53:31.351531 21706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:31.352964 21691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:53:31.353124 21691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "9ac614ef30c048558e64976e05863a4d"
format_stamp: "Formatted at 2026-09-29 02:52:45 on dist-test-slave-t6c1"
server_key: "d9869e03b78c2639426a5413b2197861"
server_key_iv: "dc8a3a4e60eb6a63853a78991e4129c7"
server_key_version: "encryptionkey@0"
I20260929 02:53:31.354171 21691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:31.371471 21691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:31.375995 21691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:31.376286 21691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:31.377342 21691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:31.378720 21691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:31.378805 21691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:31.378885 21691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:31.378968 21691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 02:53:31.408224 21691 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:43449
I20260929 02:53:31.408289 21818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:43449 every 8 connection(s)
I20260929 02:53:31.409641 21691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:53:31.412000 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 21691
I20260929 02:53:31.412591 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:39235
--local_ip_for_outbound_sockets=127.4.197.131
--tserver_master_addrs=127.4.197.190:34891
--webserver_port=42255
--webserver_interface=127.4.197.131
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:53:31.414479 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:31.412416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:43867)
0929 02:53:31.413313 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:31.413614 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":438,"threads_started":1}
W20260929 02:53:31.415143 21819 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:34891 (0 consecutive failures): Network error: Failed to ping master at 127.4.197.190:34891: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
I20260929 02:53:31.415920 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:31.415619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:47919)
0929 02:53:31.415752 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:31.415817 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:53:31.416684 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:31.416376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:40307)
0929 02:53:31.416508 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:31.416579 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 02:53:31.416894 21819 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:53:31.727689 21822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:31.727972 21822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:31.728089 21822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:31.736200 21822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:31.736330 21822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:31.736495 21822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:53:31.746517 21822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:39235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=42255
--tserver_master_addrs=127.4.197.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:31.747860 21822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:31.749140 21822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:31.761544 21830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:31.761611 21827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:31.761636 21828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:31.762475 21822 server_base.cc:1034] running on GCE node
I20260929 02:53:31.763099 21822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:31.764067 21822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:31.765277 21822 hybrid_clock.cc:648] HybridClock initialized: now 1790650411765220 us; error 53 us; skew 500 ppm
I20260929 02:53:31.765596 21822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:31 dist-test-slave-t6c1 krb5kdc[20353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650411, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:31.768543 21822 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:53:31.769873 21822 webserver.cc:460] Webserver started at http://127.4.197.131:42255/ using document root <none> and password file <none>
I20260929 02:53:31.770522 21822 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:31.770640 21822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:31.775996 21822 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:53:31.778788 21837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:31.780200 21822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:53:31.780354 21822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "4f8a55e7c21746629a6757313665b2ae"
format_stamp: "Formatted at 2026-09-29 02:52:45 on dist-test-slave-t6c1"
server_key: "7304392a848a5b466b20ed1df81b4aa0"
server_key_iv: "1444f133703bf2f580acfbf311c76753"
server_key_version: "encryptionkey@0"
I20260929 02:53:31.781316 21822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:31.795136 21822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:31.798647 21822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:31.798883 21822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:31.799783 21822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:31.800997 21822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:31.801086 21822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:31.801189 21822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:31.801256 21822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:31.830251 21822 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:39235
I20260929 02:53:31.830304 21949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:39235 every 8 connection(s)
I20260929 02:53:31.831739 21822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:53:31.834405 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 21822
I20260929 02:53:31.836874 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:31.834815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:39257)
0929 02:53:31.835654 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:31.835940 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":312,"threads_started":1}
W20260929 02:53:31.837527 21950 heartbeater.cc:644] Failed to heartbeat to 127.4.197.190:34891 (0 consecutive failures): Network error: Failed to ping master at 127.4.197.190:34891: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
I20260929 02:53:31.838322 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:31.837977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:44793)
0929 02:53:31.838119 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:31.838189 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 02:53:31.839126 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:31.838765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:42885)
0929 02:53:31.838896 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:31.839014 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 02:53:31.839352 21950 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:53:31.964315 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:31.963533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:51439)
0929 02:53:31.963997 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:31.964087 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":231,"threads_started":1}
I20260929 02:53:32.418139 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:32.417364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:57867)
0929 02:53:32.417794 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:32.417912 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":212,"threads_started":1}
I20260929 02:53:32.840744 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:32.839932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:49689)
0929 02:53:32.840413 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:32.840523 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":232,"threads_started":1}
I20260929 02:53:32.965869 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:32.965181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:38377)
0929 02:53:32.965610 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:32.965694 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":209,"threads_started":1}
I20260929 02:53:33.419714 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:33.418951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:56141)
0929 02:53:33.419443 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:33.419531 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":228,"threads_started":1}
I20260929 02:53:33.842332 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:33.841612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:42643)
0929 02:53:33.842066 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:33.842149 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":221,"threads_started":1}
I20260929 02:53:33.967576 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:33.966798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:57703)
0929 02:53:33.967285 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:33.967382 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":262,"threads_started":1}
I20260929 02:53:34.421195 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:34.420556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:50831)
0929 02:53:34.420961 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:34.421037 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":211,"threads_started":1}
I20260929 02:53:34.844012 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:34.843274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:35379)
0929 02:53:34.843748 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:34.843839 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260929 02:53:34.969280 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:34.968492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:58975)
0929 02:53:34.968963 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:34.969056 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":235,"threads_started":1}
I20260929 02:53:35.423105 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:35.422214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:39871)
0929 02:53:35.422729 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:35.422825 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":267,"threads_started":1}
I20260929 02:53:35.845671 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:35.844929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:59749)
0929 02:53:35.845398 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:35.845483 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":247,"threads_started":1}
I20260929 02:53:35.971410 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:35.970396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:54251)
0929 02:53:35.971056 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:35.971172 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":315,"threads_started":1}
I20260929 02:53:36.424639 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:36.423918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:50421)
0929 02:53:36.424378 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:36.424461 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":227,"threads_started":1}
I20260929 02:53:36.847496 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:36.846690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:39131)
0929 02:53:36.847210 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:36.847299 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":286,"threads_started":1}
I20260929 02:53:36.973219 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:36.972378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:53311)
0929 02:53:36.972878 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:36.972993 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":252,"threads_started":1}
I20260929 02:53:37.426244 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:37.425538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:51953)
0929 02:53:37.425985 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:37.426080 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":223,"threads_started":1}
I20260929 02:53:37.849186 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:37.848392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:46653)
0929 02:53:37.848893 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:37.849003 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":236,"threads_started":1}
I20260929 02:53:37.975095 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:37.974232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:60353)
0929 02:53:37.974743 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:37.974823 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":207,"threads_started":1}
I20260929 02:53:38.428087 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:38.427274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:50613)
0929 02:53:38.427772 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:38.427875 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":211,"threads_started":1}
I20260929 02:53:38.851029 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:38.850121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:59449)
0929 02:53:38.850666 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:38.850755 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":271,"threads_started":1}
I20260929 02:53:38.976613 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:38.975910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:56581)
0929 02:53:38.976366 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:38.976450 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":225,"threads_started":1}
I20260929 02:53:39.429814 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:39.429080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:36025)
0929 02:53:39.429578 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:39.429654 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":287,"threads_started":1}
I20260929 02:53:39.852842 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:39.851986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:45871)
0929 02:53:39.852548 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:39.852653 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":300,"threads_started":1}
I20260929 02:53:39.978248 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:39.977487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:57267)
0929 02:53:39.977987 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:39.978072 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":236,"threads_started":1}
I20260929 02:53:40.431499 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:40.430679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:41789)
0929 02:53:40.431235 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:40.431314 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":306,"threads_started":1}
I20260929 02:53:40.854544 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:40.853745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:40703)
0929 02:53:40.854278 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:40.854366 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":260,"threads_started":1}
I20260929 02:53:40.979883 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:40.979167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:44297)
0929 02:53:40.979626 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:40.979710 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260929 02:53:41.433377 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:41.432550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:34493)
0929 02:53:41.433088 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:41.433193 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":305,"threads_started":1}
I20260929 02:53:41.856400 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:41.855592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:48911)
0929 02:53:41.856102 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:41.856196 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":278,"threads_started":1}
I20260929 02:53:41.981451 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:41.980760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:35823)
0929 02:53:41.981166 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:41.981266 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":200,"threads_started":1}
I20260929 02:53:42.434894 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:42.434238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:58923)
0929 02:53:42.434624 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:42.434714 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":187,"threads_started":1}
I20260929 02:53:42.857995 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:42.857254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:36767)
0929 02:53:42.857707 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:42.857804 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
I20260929 02:53:42.983084 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:42.982355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:47345)
0929 02:53:42.982767 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:42.982850 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":201,"threads_started":1}
I20260929 02:53:43.436532 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:43.435820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:33885)
0929 02:53:43.436274 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:43.436371 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
I20260929 02:53:43.859651 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:43.858960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:58655)
0929 02:53:43.859407 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:43.859494 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260929 02:53:43.984877 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:43.984054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:33631)
0929 02:53:43.984575 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:43.984676 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":248,"threads_started":1}
I20260929 02:53:44.438086 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:44.437365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:59107)
0929 02:53:44.437813 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:44.437908 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":223,"threads_started":1}
I20260929 02:53:44.861322 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:44.860503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:45943)
0929 02:53:44.861012 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:44.861121 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":230,"threads_started":1}
I20260929 02:53:44.986620 21993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:44.985824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:51217)
0929 02:53:44.986316 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:44.986436 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":248,"threads_started":1}
I20260929 02:53:45.439630 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:45.438956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:59767)
0929 02:53:45.439370 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:45.439451 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":199,"threads_started":1}
I20260929 02:53:45.862890 21995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:45.862187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:42629)
0929 02:53:45.862621 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:45.862707 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260929 02:53:45.988541 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:45.987773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.129:57843)
0929 02:53:45.988246 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:45.988357 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":249,"threads_started":1}
I20260929 02:53:46.441517 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:46.440736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.130:48999)
0929 02:53:46.441191 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:46.441308 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":219,"threads_started":1}
I20260929 02:53:46.845247 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 21519
I20260929 02:53:46.865063 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:46.864150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34891 (local address 127.4.197.131:32969)
0929 02:53:46.864723 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:46.864853 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":270,"threads_started":1}
I20260929 02:53:46.867623 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:46.846675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.4.197.129:41597)
0929 02:53:46.847303 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:46.847316 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:46.847355 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:53:46.848961 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:46.848971 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:46.849419 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:46.849728 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:46.851115 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:46.852444 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:46.853787 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:46.854535 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:46.854689 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:46.860766 (+ 6077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:46.860774 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:46.860792 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:46.860834 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:46.863997 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:46.864870 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:46.864878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:46.864885 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:46.865066 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:46.865497 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:46.865501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:46.865503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:46.866157 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:46.866430 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:46.866809 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 02:53:46.867333 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":323,"threads_started":1}
I20260929 02:53:46.869220 21683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:46.953629 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 21519
I20260929 02:53:46.967401 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 21691
I20260929 02:53:46.984812 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:46.968256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.4.197.130:43449)
0929 02:53:46.968757 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:46.968771 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:46.968808 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:53:46.970324 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:46.970332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:46.970748 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:46.971088 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:46.972510 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:46.973726 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:46.975018 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:46.975718 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:46.975913 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:46.978368 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:46.978376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:46.978392 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:46.978431 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:46.981117 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:46.981776 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:46.981783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:46.981789 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:46.982001 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:46.982547 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:46.982551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:46.982553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:46.983257 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:46.983562 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:46.983998 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 02:53:46.984496 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":226,"threads_started":1}
I20260929 02:53:46.986373 21814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:47.069691 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 21691
I20260929 02:53:47.084151 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 21822
I20260929 02:53:47.101668 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:47.085137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.4.197.131:39235)
0929 02:53:47.085802 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:47.085817 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:47.085860 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:53:47.087595 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:47.087605 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:47.088101 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:47.088498 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:47.089760 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:47.091052 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:47.092352 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:47.093064 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:47.093229 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:47.095877 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:47.095885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:47.095902 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:47.095948 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:47.098524 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:47.099241 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:47.099246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:47.099253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:47.099414 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:47.099736 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:47.099739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:47.099741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:47.100261 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:47.100484 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:47.100922 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 02:53:47.101345 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":311,"threads_started":1}
I20260929 02:53:47.103128 21945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:47.186363 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 21822
2026-09-29T02:53:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64431 ms)
[----------] 30 tests from SecurityITest (289161 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:53:47 dist-test-slave-t6c1 krb5kdc[22007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:53:47 dist-test-slave-t6c1 krb5kdc[22007](info): set up 2 sockets
Sep 29 02:53:47 dist-test-slave-t6c1 krb5kdc[22007](info): commencing operation
krb5kdc: starting...
W20260929 02:53:48.202839 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:53:48 dist-test-slave-t6c1 krb5kdc[22007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:53:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:53:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:53:48.432337 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34389
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:34389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:53:48.754789 22020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:48.755177 22020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:48.755337 22020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:48.762881 22020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:53:48.763062 22020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:48.763142 22020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:48.763217 22020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:53:48.763289 22020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:53:48.773713 22020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:34389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:48.775058 22020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:48.776402 22020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:48.788982 22026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:48.788993 22028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:48.789184 22025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:48.789674 22020 server_base.cc:1034] running on GCE node
I20260929 02:53:48.790513 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:48.792106 22020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:48.793335 22020 hybrid_clock.cc:648] HybridClock initialized: now 1790650428793265 us; error 34 us; skew 500 ppm
I20260929 02:53:48.793723 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:48 dist-test-slave-t6c1 krb5kdc[22007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650428, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:48.796984 22020 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:53:48.798547 22020 webserver.cc:460] Webserver started at http://127.4.197.190:45789/ using document root <none> and password file <none>
I20260929 02:53:48.799326 22020 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:48.799432 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:48.799695 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:48.802547 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "31bcdb9835b84f09b4368c3d72b4350c"
format_stamp: "Formatted at 2026-09-29 02:53:48 on dist-test-slave-t6c1"
server_key: "926f66d6b2c344521595fde2a70fb030"
server_key_iv: "35c512d3c4f85cd4f9badf10de6ca1a5"
server_key_version: "encryptionkey@0"
I20260929 02:53:48.803347 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "31bcdb9835b84f09b4368c3d72b4350c"
format_stamp: "Formatted at 2026-09-29 02:53:48 on dist-test-slave-t6c1"
server_key: "926f66d6b2c344521595fde2a70fb030"
server_key_iv: "35c512d3c4f85cd4f9badf10de6ca1a5"
server_key_version: "encryptionkey@0"
I20260929 02:53:48.808125 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:53:48.811317 22035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:48.812554 22020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:53:48.812705 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "31bcdb9835b84f09b4368c3d72b4350c"
format_stamp: "Formatted at 2026-09-29 02:53:48 on dist-test-slave-t6c1"
server_key: "926f66d6b2c344521595fde2a70fb030"
server_key_iv: "35c512d3c4f85cd4f9badf10de6ca1a5"
server_key_version: "encryptionkey@0"
I20260929 02:53:48.812898 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:48.827780 22020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:48.831987 22020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:48.832247 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:48.849972 22020 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:34389
I20260929 02:53:48.850023 22086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:34389 every 8 connection(s)
I20260929 02:53:48.851500 22020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:53:48.853806 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 22020
I20260929 02:53:48.853997 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:53:48.854664 4886 external_mini_cluster.cc:1428] Setting key b8454cfc98e96e783fbfd7c88d259a1a
I20260929 02:53:48.855777 22087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:48.870903 22087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c: Bootstrap starting.
Sep 29 02:53:48 dist-test-slave-t6c1 krb5kdc[22007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:48.874715 22087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:48.876080 22087 log.cc:824] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:48.879629 22087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c: No bootstrap required, opened a new log
I20260929 02:53:48.881433 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:48.856747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41960 (local address 127.4.197.190:34389)
0929 02:53:48.857733 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:48.857750 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:48.858169 (+ 419us) server_negotiation.cc:407] Connection header received
0929 02:53:48.860929 (+ 2760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:48.860957 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:48.861508 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:48.861907 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:48.863833 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:48.865154 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:48.866959 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:48.867746 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:48.867882 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:48.872620 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:48.872631 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:48.872652 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:48.872709 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:48.876544 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:48.877512 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:48.877521 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:48.877529 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:48.877749 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:48.878170 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:48.878175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:48.878178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:48.878909 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:48.879221 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:48.879650 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 02:53:48.880087 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":402,"threads_started":1}
I20260929 02:53:48.888119 22087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcdb9835b84f09b4368c3d72b4350c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34389 } }
I20260929 02:53:48.888444 22087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:48.888515 22087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31bcdb9835b84f09b4368c3d72b4350c, State: Initialized, Role: FOLLOWER
I20260929 02:53:48.889008 22087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcdb9835b84f09b4368c3d72b4350c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34389 } }
I20260929 02:53:48.889202 22087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:53:48.889293 22087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:53:48.889410 22087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:48.891463 22087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcdb9835b84f09b4368c3d72b4350c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34389 } }
I20260929 02:53:48.891888 22087 leader_election.cc:302] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31bcdb9835b84f09b4368c3d72b4350c; no voters:
I20260929 02:53:48.892377 22087 leader_election.cc:288] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:53:48.892511 22092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:53:48.893095 22092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [term 1 LEADER]: Becoming Leader. State: Replica: 31bcdb9835b84f09b4368c3d72b4350c, State: Running, Role: LEADER
I20260929 02:53:48.893600 22092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcdb9835b84f09b4368c3d72b4350c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34389 } }
I20260929 02:53:48.894395 22087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:53:48.896183 22093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31bcdb9835b84f09b4368c3d72b4350c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcdb9835b84f09b4368c3d72b4350c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34389 } } }
I20260929 02:53:48.896574 22093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:48.896427 22094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31bcdb9835b84f09b4368c3d72b4350c. Latest consensus state: current_term: 1 leader_uuid: "31bcdb9835b84f09b4368c3d72b4350c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcdb9835b84f09b4368c3d72b4350c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34389 } } }
I20260929 02:53:48.896697 22094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:48.898010 22098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:53:48.903168 22098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:53:48.911943 22098 catalog_manager.cc:1347] Generated new cluster ID: 4b3598910ead4d00832e625c104b58e0
I20260929 02:53:48.912056 22098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:53:48.929128 22098 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:53:48.930471 22098 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:53:48.936965 22098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c: Generated new TSK 0
I20260929 02:53:48.937806 22098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:53:49.010413 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34389
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:53:49.355504 22113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:49.355798 22113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:49.355909 22113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:49.363278 22113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:49.363389 22113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:49.363508 22113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:53:49.373103 22113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:49.374320 22113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:49.375598 22113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:49.388422 22119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:49.388391 22118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:49.388499 22121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:49.389411 22113 server_base.cc:1034] running on GCE node
I20260929 02:53:49.390007 22113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:49.390897 22113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:49.392122 22113 hybrid_clock.cc:648] HybridClock initialized: now 1790650429392062 us; error 55 us; skew 500 ppm
I20260929 02:53:49.392441 22113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:49 dist-test-slave-t6c1 krb5kdc[22007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650429, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:49.395529 22113 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:53:49.396984 22113 webserver.cc:460] Webserver started at http://127.4.197.129:39965/ using document root <none> and password file <none>
I20260929 02:53:49.397686 22113 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:49.397785 22113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:49.398036 22113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:49.400866 22113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "a75aa73e1ab847d1943d881186533cc1"
format_stamp: "Formatted at 2026-09-29 02:53:49 on dist-test-slave-t6c1"
server_key: "1d2c30d6df2feca6096817532887c7fd"
server_key_iv: "16f91767286a60d6180a8a46f454c996"
server_key_version: "encryptionkey@0"
I20260929 02:53:49.401587 22113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "a75aa73e1ab847d1943d881186533cc1"
format_stamp: "Formatted at 2026-09-29 02:53:49 on dist-test-slave-t6c1"
server_key: "1d2c30d6df2feca6096817532887c7fd"
server_key_iv: "16f91767286a60d6180a8a46f454c996"
server_key_version: "encryptionkey@0"
I20260929 02:53:49.407335 22113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:53:49.410624 22128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:49.411971 22113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:53:49.412122 22113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "a75aa73e1ab847d1943d881186533cc1"
format_stamp: "Formatted at 2026-09-29 02:53:49 on dist-test-slave-t6c1"
server_key: "1d2c30d6df2feca6096817532887c7fd"
server_key_iv: "16f91767286a60d6180a8a46f454c996"
server_key_version: "encryptionkey@0"
I20260929 02:53:49.412300 22113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:49.437917 22113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:49.442116 22113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:49.442389 22113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:49.443444 22113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:49.444782 22113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:49.444859 22113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:49.444974 22113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:49.445029 22113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:49.474614 22113 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:44971
I20260929 02:53:49.474654 22240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:44971 every 8 connection(s)
I20260929 02:53:49.476100 22113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:53:49.477432 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 22113
I20260929 02:53:49.477643 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:53:49.478449 4886 external_mini_cluster.cc:1428] Setting key 37061afcf505c68c23423d7902adedd7
Sep 29 02:53:49 dist-test-slave-t6c1 krb5kdc[22007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650429, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:49.499364 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:49.479102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:55089 (local address 127.4.197.190:34389)
0929 02:53:49.479621 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:49.479626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:49.480886 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 02:53:49.483653 (+ 2767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:49.483657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:49.483729 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:49.483814 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:49.485895 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:49.486661 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:49.488966 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:49.489373 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:49.489475 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:49.493392 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:49.493398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:49.493402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:49.493439 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:49.496205 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:49.497099 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:49.497103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:49.497106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:49.497227 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:49.497671 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:49.497674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:49.497676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:49.497949 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:49.498059 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:49.498885 (+ 826us) server_negotiation.cc:299] Negotiation successful
0929 02:53:49.499109 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":245,"threads_started":1}
I20260929 02:53:49.500777 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:49.479777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34389 (local address 127.4.197.129:55089)
0929 02:53:49.480692 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:49.480762 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:49.483298 (+ 2536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:49.484102 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:49.484114 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:49.484876 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:49.485686 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:49.485711 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:49.486830 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:49.486835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:49.488810 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:49.488826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:49.489598 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:49.489604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:49.489889 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:49.490668 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:53:49.490692 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:49.493134 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:49.496348 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:49.496358 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:49.496376 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:49.496984 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:49.497315 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:49.497320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:49.497323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:49.497576 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:49.498068 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:49.498075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:49.498498 (+ 423us) client_negotiation.cc:769] Sending connection context
0929 02:53:49.498781 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 02:53:49.499370 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":363,"threads_started":1}
I20260929 02:53:49.502437 22241 heartbeater.cc:347] Connected to a master server at 127.4.197.190:34389
I20260929 02:53:49.502718 22241 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:49.503473 22241 heartbeater.cc:511] Master 127.4.197.190:34389 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:53:49.505513 22052 ts_manager.cc:194] Registered new tserver with Master: a75aa73e1ab847d1943d881186533cc1 (127.4.197.129:44971)
I20260929 02:53:49.507009 22052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:55089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:53:49.534070 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34389
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:53:49.850206 22247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:49.850495 22247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:49.850629 22247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:49.857954 22247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:49.858083 22247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:49.858246 22247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:53:49.868243 22247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:49.869527 22247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:49.870841 22247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:49.883705 22252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:49.883742 22255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:49.883749 22253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:49.884510 22247 server_base.cc:1034] running on GCE node
I20260929 02:53:49.885077 22247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:49.886066 22247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:49.887264 22247 hybrid_clock.cc:648] HybridClock initialized: now 1790650429887211 us; error 40 us; skew 500 ppm
I20260929 02:53:49.887566 22247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:49 dist-test-slave-t6c1 krb5kdc[22007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650429, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:49.890619 22247 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:53:49.892032 22247 webserver.cc:460] Webserver started at http://127.4.197.130:43013/ using document root <none> and password file <none>
I20260929 02:53:49.892684 22247 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:49.892800 22247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:49.893086 22247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:49.896063 22247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "3775ee775fcd4faf9fc3eec482b56296"
format_stamp: "Formatted at 2026-09-29 02:53:49 on dist-test-slave-t6c1"
server_key: "d27d9bd67f6edf0002fcb5400260c316"
server_key_iv: "b6da5ed4221994520e1c6129e0199c9d"
server_key_version: "encryptionkey@0"
I20260929 02:53:49.896791 22247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "3775ee775fcd4faf9fc3eec482b56296"
format_stamp: "Formatted at 2026-09-29 02:53:49 on dist-test-slave-t6c1"
server_key: "d27d9bd67f6edf0002fcb5400260c316"
server_key_iv: "b6da5ed4221994520e1c6129e0199c9d"
server_key_version: "encryptionkey@0"
I20260929 02:53:49.901616 22247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:53:49.904860 22262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:49.906132 22247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:53:49.906298 22247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "3775ee775fcd4faf9fc3eec482b56296"
format_stamp: "Formatted at 2026-09-29 02:53:49 on dist-test-slave-t6c1"
server_key: "d27d9bd67f6edf0002fcb5400260c316"
server_key_iv: "b6da5ed4221994520e1c6129e0199c9d"
server_key_version: "encryptionkey@0"
I20260929 02:53:49.906461 22247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:49.920450 22247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:49.924115 22247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:49.924371 22247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:49.925252 22247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:49.926512 22247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:49.926589 22247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:49.926712 22247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:49.926767 22247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:49.956943 22247 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:36557
I20260929 02:53:49.956983 22374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:36557 every 8 connection(s)
I20260929 02:53:49.958277 22247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:53:49.966492 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 22247
I20260929 02:53:49.966642 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:53:49.967195 4886 external_mini_cluster.cc:1428] Setting key f857b1fc5544f52a28d69f6a284ae93c
Sep 29 02:53:49 dist-test-slave-t6c1 krb5kdc[22007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650429, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:49.978781 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:49.960901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:51879 (local address 127.4.197.190:34389)
0929 02:53:49.961077 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:49.961082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:49.962301 (+ 1219us) server_negotiation.cc:407] Connection header received
0929 02:53:49.964295 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:49.964300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:49.964394 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:49.964488 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:49.966064 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:49.966822 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:49.968932 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:49.969443 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:49.969556 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:49.973192 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:49.973197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:49.973200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:49.973234 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:49.975715 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:49.976458 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:49.976462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:49.976465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:49.976593 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:49.977016 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:49.977020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:49.977022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:49.977303 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:49.977439 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:49.978187 (+ 748us) server_negotiation.cc:299] Negotiation successful
0929 02:53:49.978433 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:53:49.979836 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:49.961440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34389 (local address 127.4.197.130:51879)
0929 02:53:49.962137 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:49.962182 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:49.964000 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:49.964717 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:49.964726 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:49.965294 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:49.965911 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:49.965929 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:49.967005 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:49.967011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:49.968776 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:49.968791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:49.969665 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:49.969672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:49.969908 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:49.970585 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:53:49.970608 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:49.972958 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:49.975868 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:49.975878 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:49.975896 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:49.976318 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:49.976698 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:49.976703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:49.976706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:49.976902 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:49.977428 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:49.977436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:49.977814 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 02:53:49.978059 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 02:53:49.978643 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":274,"threads_started":1}
I20260929 02:53:49.981345 22375 heartbeater.cc:347] Connected to a master server at 127.4.197.190:34389
I20260929 02:53:49.981642 22375 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:49.982383 22375 heartbeater.cc:511] Master 127.4.197.190:34389 requested a full tablet report, sending...
I20260929 02:53:49.983927 22052 ts_manager.cc:194] Registered new tserver with Master: 3775ee775fcd4faf9fc3eec482b56296 (127.4.197.130:36557)
I20260929 02:53:49.984782 22052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:51879
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:53:50.018155 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34389
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:53:50.342211 22380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:50.342519 22380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:50.342646 22380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:50.350021 22380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:50.350164 22380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:50.350323 22380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:53:50.360394 22380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:34389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:50.361696 22380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:50.363137 22380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:50.377103 22388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:50.377068 22386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:50.377137 22385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:50.377946 22380 server_base.cc:1034] running on GCE node
I20260929 02:53:50.378650 22380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:50.379835 22380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:50.381083 22380 hybrid_clock.cc:648] HybridClock initialized: now 1790650430381023 us; error 48 us; skew 500 ppm
I20260929 02:53:50.381450 22380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:50 dist-test-slave-t6c1 krb5kdc[22007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650430, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:50.384886 22380 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:53:50.386619 22380 webserver.cc:460] Webserver started at http://127.4.197.131:43323/ using document root <none> and password file <none>
I20260929 02:53:50.387408 22380 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:50.387545 22380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:50.387890 22380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:50.391166 22380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "7cd2b625e8a94f6d8c7ab7184e878355"
format_stamp: "Formatted at 2026-09-29 02:53:50 on dist-test-slave-t6c1"
server_key: "d157ad038f8304ba3597378ddefff7da"
server_key_iv: "ee7395c5cb4c18fa0f60abfc6086e801"
server_key_version: "encryptionkey@0"
I20260929 02:53:50.392171 22380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "7cd2b625e8a94f6d8c7ab7184e878355"
format_stamp: "Formatted at 2026-09-29 02:53:50 on dist-test-slave-t6c1"
server_key: "d157ad038f8304ba3597378ddefff7da"
server_key_iv: "ee7395c5cb4c18fa0f60abfc6086e801"
server_key_version: "encryptionkey@0"
I20260929 02:53:50.397646 22380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:53:50.401341 22395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:50.402655 22380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:53:50.402832 22380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "7cd2b625e8a94f6d8c7ab7184e878355"
format_stamp: "Formatted at 2026-09-29 02:53:50 on dist-test-slave-t6c1"
server_key: "d157ad038f8304ba3597378ddefff7da"
server_key_iv: "ee7395c5cb4c18fa0f60abfc6086e801"
server_key_version: "encryptionkey@0"
I20260929 02:53:50.403079 22380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:50.425107 22380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:50.429255 22380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:50.429522 22380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:50.430507 22380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:50.431916 22380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:50.431999 22380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:50.432116 22380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:50.432168 22380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:50.462262 22380 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:41601
I20260929 02:53:50.462318 22507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:41601 every 8 connection(s)
I20260929 02:53:50.463603 22380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:53:50.471566 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 22380
I20260929 02:53:50.471756 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:53:50.472278 4886 external_mini_cluster.cc:1428] Setting key fb7d8729a5a92e901fbd1da7f4d5ddf0
Sep 29 02:53:50 dist-test-slave-t6c1 krb5kdc[22007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650430, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:50.483312 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.466144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:35307 (local address 127.4.197.190:34389)
0929 02:53:50.466301 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:50.466306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:50.467483 (+ 1177us) server_negotiation.cc:407] Connection header received
0929 02:53:50.469303 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:50.469309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:50.469385 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:50.469473 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:50.470992 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.472070 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.473835 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.474411 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.474528 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.477771 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:50.477777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:50.477780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:50.477811 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:50.480420 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:50.481070 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:50.481075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:50.481079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:50.481196 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:50.481589 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:50.481594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:50.481597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:50.481859 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:50.481988 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:50.482723 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 02:53:50.482986 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:53:50.484180 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.466608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:34389 (local address 127.4.197.131:35307)
0929 02:53:50.467316 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:50.467366 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:50.469013 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:50.469709 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:50.469717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:50.470237 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:50.470803 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.470826 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.472219 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.472223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.473688 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.473701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.474624 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.474633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.474858 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.475518 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:53:50.475542 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:50.477578 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:50.480546 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:50.480555 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:50.480570 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:50.480962 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:50.481295 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:50.481300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:50.481304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:50.481489 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:50.481975 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:50.481982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:50.482298 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 02:53:50.482603 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 02:53:50.483134 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":292,"threads_started":1}
I20260929 02:53:50.485385 22508 heartbeater.cc:347] Connected to a master server at 127.4.197.190:34389
I20260929 02:53:50.485638 22508 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:50.486387 22508 heartbeater.cc:511] Master 127.4.197.190:34389 requested a full tablet report, sending...
I20260929 02:53:50.487920 22052 ts_manager.cc:194] Registered new tserver with Master: 7cd2b625e8a94f6d8c7ab7184e878355 (127.4.197.131:41601)
I20260929 02:53:50.488727 22052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:35307
I20260929 02:53:50.500108 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:53:50.509868 22241 heartbeater.cc:503] Master 127.4.197.190:34389 was elected leader, sending a full tablet report...
I20260929 02:53:50.516630 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.504522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.4.197.190:34389)
0929 02:53:50.504683 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:50.504688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:50.505026 (+ 338us) server_negotiation.cc:407] Connection header received
0929 02:53:50.505158 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:50.505161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:50.505221 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:50.505302 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:50.506543 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.507314 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.508785 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.509187 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.509276 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.511699 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:50.511703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:50.511706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:50.511735 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:50.514123 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:50.514695 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:50.514698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:50.514700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:50.514820 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:50.515225 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:50.515228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:50.515230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:50.515633 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:50.515849 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:50.516114 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 02:53:50.516331 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:53:50.520263 22052 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:53:50.527073 22052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:53:50.549384 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.539264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:36557 (local address 127.0.0.1:40434)
0929 02:53:50.540009 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:50.540060 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:50.540217 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:50.541499 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:50.541507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:50.541538 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:50.541959 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.541973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.544142 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.544148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.546468 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.546486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.548415 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.548422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.548613 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.548670 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 02:53:50.548778 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 02:53:50.548979 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":675,"threads_started":1}
I20260929 02:53:50.550709 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.539403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.4.197.130:36557)
0929 02:53:50.540805 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:50.540813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:50.540844 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:53:50.540948 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:50.540957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:50.541236 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:50.542175 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:50.542415 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.543987 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.546686 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.548174 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.549929 (+ 1755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.550129 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:50.550207 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 02:53:50.550390 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1232,"thread_start_us":1097,"threads_started":1}
I20260929 02:53:50.551069 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.539448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:41601 (local address 127.0.0.1:46244)
0929 02:53:50.542079 (+ 2631us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:50.542095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:50.542197 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:50.543013 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:50.543018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:50.543042 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:50.543425 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.543437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.545520 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.545525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.547574 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.547590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.550130 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.550136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.550306 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.550339 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:53:50.550412 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 02:53:50.550586 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2512,"thread_start_us":2361,"threads_started":1}
I20260929 02:53:50.551640 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.539667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.4.197.131:41601)
0929 02:53:50.541822 (+ 2155us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:50.541831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:50.542307 (+ 476us) server_negotiation.cc:407] Connection header received
0929 02:53:50.542407 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:50.542414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:50.542659 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:50.542849 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:50.543800 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.545339 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.547753 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.549144 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.549256 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.551058 (+ 1802us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:50.551138 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 02:53:50.551322 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1967,"thread_start_us":1812,"threads_started":1}
I20260929 02:53:50.555181 22310 tablet_service.cc:1467] Processing CreateTablet for tablet 15680597349c4f3a85da45d261d9cc54 (DEFAULT_TABLE table=test-table [id=b51735b458e74d11a25032a5bca1006f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:50.555943 22443 tablet_service.cc:1467] Processing CreateTablet for tablet 15680597349c4f3a85da45d261d9cc54 (DEFAULT_TABLE table=test-table [id=b51735b458e74d11a25032a5bca1006f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:50.556846 22310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15680597349c4f3a85da45d261d9cc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:50.557497 22443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15680597349c4f3a85da45d261d9cc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:50.558188 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.542147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:44971 (local address 127.0.0.1:47606)
0929 02:53:50.549659 (+ 7512us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:50.549677 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:50.549776 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:50.552936 (+ 3160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:50.552940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:50.552966 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:50.553128 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.553138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.554579 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.554584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.555869 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.555883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.557456 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.557461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.557638 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.557669 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:53:50.557760 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:53:50.557909 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7425,"thread_start_us":9863,"threads_started":1}
I20260929 02:53:50.558296 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.542543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.4.197.129:44971)
0929 02:53:50.552331 (+ 9788us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:50.552339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:50.552367 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:53:50.552476 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:50.552483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:50.552743 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:50.552958 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:50.553299 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.554440 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.556058 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.557299 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.557428 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.557666 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:50.557825 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 02:53:50.557998 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9644,"thread_start_us":9329,"threads_started":1}
I20260929 02:53:50.561226 22176 tablet_service.cc:1467] Processing CreateTablet for tablet 15680597349c4f3a85da45d261d9cc54 (DEFAULT_TABLE table=test-table [id=b51735b458e74d11a25032a5bca1006f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:50.562516 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15680597349c4f3a85da45d261d9cc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:50.573608 22526 tablet_bootstrap.cc:492] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: Bootstrap starting.
I20260929 02:53:50.574573 22527 tablet_bootstrap.cc:492] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: Bootstrap starting.
I20260929 02:53:50.575789 22528 tablet_bootstrap.cc:492] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: Bootstrap starting.
I20260929 02:53:50.578399 22526 tablet_bootstrap.cc:654] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:50.578722 22527 tablet_bootstrap.cc:654] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:50.579245 22528 tablet_bootstrap.cc:654] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:50.579617 22526 log.cc:824] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:50.580037 22527 log.cc:824] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:50.580420 22528 log.cc:824] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:50.582288 22526 tablet_bootstrap.cc:492] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: No bootstrap required, opened a new log
I20260929 02:53:50.582561 22526 ts_tablet_manager.cc:1397] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:53:50.583086 22528 tablet_bootstrap.cc:492] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: No bootstrap required, opened a new log
I20260929 02:53:50.583376 22528 ts_tablet_manager.cc:1397] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:53:50.584780 22527 tablet_bootstrap.cc:492] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: No bootstrap required, opened a new log
I20260929 02:53:50.585086 22527 ts_tablet_manager.cc:1397] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 02:53:50.590459 22526 raft_consensus.cc:348] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } }
I20260929 02:53:50.590903 22526 raft_consensus.cc:374] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:50.591044 22526 raft_consensus.cc:729] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3775ee775fcd4faf9fc3eec482b56296, State: Initialized, Role: FOLLOWER
I20260929 02:53:50.591826 22526 consensus_queue.cc:260] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } }
I20260929 02:53:50.593976 22375 heartbeater.cc:503] Master 127.4.197.190:34389 was elected leader, sending a full tablet report...
I20260929 02:53:50.593899 22528 raft_consensus.cc:348] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } }
I20260929 02:53:50.594424 22528 raft_consensus.cc:374] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:50.594579 22528 raft_consensus.cc:729] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75aa73e1ab847d1943d881186533cc1, State: Initialized, Role: FOLLOWER
I20260929 02:53:50.595454 22528 consensus_queue.cc:260] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } }
I20260929 02:53:50.595844 22527 raft_consensus.cc:348] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } }
I20260929 02:53:50.596331 22527 raft_consensus.cc:374] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:50.596474 22527 raft_consensus.cc:729] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd2b625e8a94f6d8c7ab7184e878355, State: Initialized, Role: FOLLOWER
I20260929 02:53:50.597074 22527 consensus_queue.cc:260] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } }
I20260929 02:53:50.597440 22528 ts_tablet_manager.cc:1428] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 02:53:50.598170 22508 heartbeater.cc:503] Master 127.4.197.190:34389 was elected leader, sending a full tablet report...
I20260929 02:53:50.598898 22527 ts_tablet_manager.cc:1428] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 02:53:50.599651 22526 ts_tablet_manager.cc:1428] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: Time spent starting tablet: real 0.017s user 0.007s sys 0.004s
W20260929 02:53:50.710206 22376 tablet.cc:2369] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:53:50.715154 22509 tablet.cc:2369] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:53:50.728521 22242 tablet.cc:2369] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:53:50.896160 22534 raft_consensus.cc:482] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:53:50.896471 22534 raft_consensus.cc:504] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } }
I20260929 02:53:50.898258 22534 leader_election.cc:288] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3775ee775fcd4faf9fc3eec482b56296 (127.4.197.130:36557), a75aa73e1ab847d1943d881186533cc1 (127.4.197.129:44971)
I20260929 02:53:50.905125 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.898765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:50369 (local address 127.4.197.129:44971)
0929 02:53:50.898991 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:50.898997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:50.899088 (+ 91us) server_negotiation.cc:407] Connection header received
0929 02:53:50.899289 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:50.899295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:50.899373 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:50.899451 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:50.899911 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.900877 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.903188 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.904362 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.904519 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.904591 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:50.904812 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 02:53:50.904909 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:53:50.905216 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.898571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:44971 (local address 127.4.197.131:50369)
0929 02:53:50.898973 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:50.898992 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:50.899141 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:50.899500 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:50.899505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:50.899536 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:50.899770 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.899779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.901023 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.901027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.903024 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.903039 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.904498 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.904503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.904646 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.904680 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:53:50.904779 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:53:50.904920 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"spinlock_wait_cycles":24448}
I20260929 02:53:50.905468 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.898750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:43291 (local address 127.4.197.130:36557)
0929 02:53:50.898974 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:50.898980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:50.899559 (+ 579us) server_negotiation.cc:407] Connection header received
0929 02:53:50.899679 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:50.899682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:50.899742 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:50.899810 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:50.900318 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.901202 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.903407 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.904563 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.904696 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.904746 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:50.905052 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 02:53:50.905235 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:53:50.905659 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.898711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:36557 (local address 127.4.197.131:43291)
0929 02:53:50.899452 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:50.899467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:50.899572 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:50.899846 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:50.899858 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:50.899883 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:50.900179 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.900187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.901338 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.901343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.903263 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.903273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.904735 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.904740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.904911 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.904939 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:53:50.905018 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:53:50.905131 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":424,"threads_started":1}
I20260929 02:53:50.906183 22196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15680597349c4f3a85da45d261d9cc54" candidate_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75aa73e1ab847d1943d881186533cc1" is_pre_election: true
I20260929 02:53:50.906564 22532 raft_consensus.cc:482] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:53:50.906699 22196 raft_consensus.cc:2454] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cd2b625e8a94f6d8c7ab7184e878355 in term 0.
I20260929 02:53:50.906595 22330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15680597349c4f3a85da45d261d9cc54" candidate_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3775ee775fcd4faf9fc3eec482b56296" is_pre_election: true
I20260929 02:53:50.906903 22532 raft_consensus.cc:504] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } }
I20260929 02:53:50.907416 22398 leader_election.cc:302] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cd2b625e8a94f6d8c7ab7184e878355, a75aa73e1ab847d1943d881186533cc1; no voters:
I20260929 02:53:50.907815 22534 raft_consensus.cc:2781] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:53:50.907761 22330 raft_consensus.cc:2454] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cd2b625e8a94f6d8c7ab7184e878355 in term 0.
I20260929 02:53:50.908044 22534 raft_consensus.cc:482] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:53:50.908179 22534 raft_consensus.cc:3037] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:50.908739 22532 leader_election.cc:288] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cd2b625e8a94f6d8c7ab7184e878355 (127.4.197.131:41601), a75aa73e1ab847d1943d881186533cc1 (127.4.197.129:44971)
I20260929 02:53:50.911535 22534 raft_consensus.cc:504] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } }
I20260929 02:53:50.912319 22534 leader_election.cc:288] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [CANDIDATE]: Term 1 election: Requested vote from peers 3775ee775fcd4faf9fc3eec482b56296 (127.4.197.130:36557), a75aa73e1ab847d1943d881186533cc1 (127.4.197.129:44971)
I20260929 02:53:50.912819 22330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15680597349c4f3a85da45d261d9cc54" candidate_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3775ee775fcd4faf9fc3eec482b56296"
I20260929 02:53:50.913033 22330 raft_consensus.cc:3037] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:50.916019 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.908967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:45963 (local address 127.4.197.131:41601)
0929 02:53:50.909182 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:50.909191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:50.909583 (+ 392us) server_negotiation.cc:407] Connection header received
0929 02:53:50.909813 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:50.909818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:50.909888 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:50.909946 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:50.910704 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.911778 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.914228 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.915132 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.915271 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.915341 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:50.915633 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 02:53:50.915779 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:53:50.915949 22196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15680597349c4f3a85da45d261d9cc54" candidate_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75aa73e1ab847d1943d881186533cc1"
I20260929 02:53:50.916167 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.908908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:41601 (local address 127.4.197.130:45963)
0929 02:53:50.909491 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:50.909510 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:50.909676 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:50.910121 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:50.910125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:50.910152 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:50.910524 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.910537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.911901 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.911906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.914053 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.914071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.915341 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.915347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.915497 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.915520 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:53:50.915598 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:53:50.915715 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":49,"thread_start_us":245,"threads_started":1}
I20260929 02:53:50.916188 22196 raft_consensus.cc:3037] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:50.916863 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.909709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:44011 (local address 127.4.197.129:44971)
0929 02:53:50.909888 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:50.909894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:50.910448 (+ 554us) server_negotiation.cc:407] Connection header received
0929 02:53:50.910618 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:50.910622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:50.910690 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:50.910770 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:50.911389 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.912338 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.914498 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.915706 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.915868 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.915936 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:50.916463 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 02:53:50.916598 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:53:50.917505 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:50.909493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:44971 (local address 127.4.197.130:44011)
0929 02:53:50.910360 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:50.910375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:50.910497 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:50.910789 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:50.910793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:50.910816 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:50.911214 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.911226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.912676 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.912682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.914336 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:50.914361 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:50.916084 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:50.916090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:50.916278 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:50.916311 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:53:50.917020 (+ 709us) client_negotiation.cc:240] Negotiation successful
0929 02:53:50.917147 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":4,"thread_start_us":544,"threads_started":1}
I20260929 02:53:50.918251 22195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15680597349c4f3a85da45d261d9cc54" candidate_uuid: "3775ee775fcd4faf9fc3eec482b56296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75aa73e1ab847d1943d881186533cc1" is_pre_election: true
I20260929 02:53:50.919637 22463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15680597349c4f3a85da45d261d9cc54" candidate_uuid: "3775ee775fcd4faf9fc3eec482b56296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" is_pre_election: true
I20260929 02:53:50.920065 22463 raft_consensus.cc:2379] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3775ee775fcd4faf9fc3eec482b56296 in current term 1: Already voted for candidate 7cd2b625e8a94f6d8c7ab7184e878355 in this term.
I20260929 02:53:50.920717 22265 leader_election.cc:302] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3775ee775fcd4faf9fc3eec482b56296; no voters: 7cd2b625e8a94f6d8c7ab7184e878355, a75aa73e1ab847d1943d881186533cc1
I20260929 02:53:50.921048 22330 raft_consensus.cc:2454] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cd2b625e8a94f6d8c7ab7184e878355 in term 1.
I20260929 02:53:50.921218 22532 raft_consensus.cc:2726] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:53:50.921468 22399 leader_election.cc:302] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3775ee775fcd4faf9fc3eec482b56296, 7cd2b625e8a94f6d8c7ab7184e878355; no voters:
I20260929 02:53:50.921525 22196 raft_consensus.cc:2454] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cd2b625e8a94f6d8c7ab7184e878355 in term 1.
I20260929 02:53:50.921834 22534 raft_consensus.cc:2781] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:53:50.922231 22534 raft_consensus.cc:686] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 1 LEADER]: Becoming Leader. State: Replica: 7cd2b625e8a94f6d8c7ab7184e878355, State: Running, Role: LEADER
I20260929 02:53:50.922667 22534 consensus_queue.cc:237] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } }
I20260929 02:53:50.927294 22051 catalog_manager.cc:5495] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cd2b625e8a94f6d8c7ab7184e878355 (127.4.197.131). New cstate: current_term: 1 leader_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cd2b625e8a94f6d8c7ab7184e878355" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 } health_report { overall_health: UNKNOWN } } }
I20260929 02:53:51.020084 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.014759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.4.197.131:41601)
0929 02:53:51.014983 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:51.014989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:51.015010 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:53:51.015095 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:51.015099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:51.015175 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:51.015296 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:53:51.015638 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:51.016454 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:51.017964 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:51.018357 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:51.018458 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:51.018990 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:53:51.019489 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:53:51.019595 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:51.019726 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 02:53:51.019791 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:53:51.028191 22196 raft_consensus.cc:1261] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1 [term 1 FOLLOWER]: Refusing update from remote peer 7cd2b625e8a94f6d8c7ab7184e878355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:53:51.028249 22330 raft_consensus.cc:1261] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 1 FOLLOWER]: Refusing update from remote peer 7cd2b625e8a94f6d8c7ab7184e878355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:53:51.029213 22534 consensus_queue.cc:1035] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3775ee775fcd4faf9fc3eec482b56296" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 36557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:53:51.029536 22538 consensus_queue.cc:1035] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a75aa73e1ab847d1943d881186533cc1" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:53:51.055946 22051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:53:51.056270 22051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:53:51.060168 22051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31bcdb9835b84f09b4368c3d72b4350c: Sending DeleteTablet for 3 replicas of tablet 15680597349c4f3a85da45d261d9cc54
I20260929 02:53:51.061480 22310 tablet_service.cc:1514] Processing DeleteTablet for tablet 15680597349c4f3a85da45d261d9cc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:51 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:40434
I20260929 02:53:51.061480 22176 tablet_service.cc:1514] Processing DeleteTablet for tablet 15680597349c4f3a85da45d261d9cc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:51 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:47606
I20260929 02:53:51.061594 22443 tablet_service.cc:1514] Processing DeleteTablet for tablet 15680597349c4f3a85da45d261d9cc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:51 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:46244
I20260929 02:53:51.063951 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 22113
I20260929 02:53:51.077358 22552 tablet_replica.cc:330] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: stopping tablet replica
I20260929 02:53:51.077905 22552 raft_consensus.cc:2229] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:53:51.078796 22552 raft_consensus.cc:2258] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:51.081993 22552 ts_tablet_manager.cc:1905] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:51.090312 22552 ts_tablet_manager.cc:1918] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:51.091054 22551 tablet_replica.cc:330] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: stopping tablet replica
I20260929 02:53:51.091589 22552 log.cc:1196] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/15680597349c4f3a85da45d261d9cc54
I20260929 02:53:51.091656 22550 tablet_replica.cc:330] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: stopping tablet replica
I20260929 02:53:51.092207 22552 ts_tablet_manager.cc:1939] T 15680597349c4f3a85da45d261d9cc54 P 7cd2b625e8a94f6d8c7ab7184e878355: Deleting consensus metadata
I20260929 02:53:51.092279 22550 raft_consensus.cc:2229] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:53:51.092753 22551 raft_consensus.cc:2229] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:53:51.093358 22550 raft_consensus.cc:2258] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:51.093888 22551 raft_consensus.cc:2258] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 02:53:51 dist-test-slave-t6c1 krb5kdc[22007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:53:51.094256 22038 catalog_manager.cc:4841] TS 7cd2b625e8a94f6d8c7ab7184e878355 (127.4.197.131:41601): tablet 15680597349c4f3a85da45d261d9cc54 (table test-table [id=b51735b458e74d11a25032a5bca1006f]) successfully deleted
I20260929 02:53:51.096326 22550 ts_tablet_manager.cc:1905] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:51.096716 22551 ts_tablet_manager.cc:1905] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:51.103583 22550 ts_tablet_manager.cc:1918] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:51.103590 22551 ts_tablet_manager.cc:1918] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:51.103682 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.065663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.4.197.129:44971)
0929 02:53:51.065861 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:51.065868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:51.079787 (+ 13919us) server_negotiation.cc:407] Connection header received
0929 02:53:51.079990 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:51.079996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:51.080097 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:51.080201 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:51.083625 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:51.084723 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:51.088780 (+ 4057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:51.089450 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:51.089554 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:51.095567 (+ 6013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:51.095578 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:51.095595 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:51.095642 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:51.099900 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:51.100804 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:51.100812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:51.100816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:51.100990 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:51.101472 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:51.101477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:51.101480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:51.101934 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:51.102239 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:51.103123 (+ 884us) server_negotiation.cc:299] Negotiation successful
0929 02:53:51.103329 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:53:51.103732 22551 log.cc:1196] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/15680597349c4f3a85da45d261d9cc54
I20260929 02:53:51.104205 22551 ts_tablet_manager.cc:1939] T 15680597349c4f3a85da45d261d9cc54 P 3775ee775fcd4faf9fc3eec482b56296: Deleting consensus metadata
I20260929 02:53:51.104458 22550 log.cc:1196] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/15680597349c4f3a85da45d261d9cc54
I20260929 02:53:51.104593 22236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:51.105962 22039 catalog_manager.cc:4841] TS 3775ee775fcd4faf9fc3eec482b56296 (127.4.197.130:36557): tablet 15680597349c4f3a85da45d261d9cc54 (table test-table [id=b51735b458e74d11a25032a5bca1006f]) successfully deleted
I20260929 02:53:51.238590 22550 ts_tablet_manager.cc:1939] T 15680597349c4f3a85da45d261d9cc54 P a75aa73e1ab847d1943d881186533cc1: Deleting consensus metadata
I20260929 02:53:51.239207 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 22113
W20260929 02:53:51.259905 22038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.197.129:44971 (error 108)
I20260929 02:53:51.260637 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 22247
I20260929 02:53:51.262285 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.261269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:44971 (local address 127.0.0.1:47614)
0929 02:53:51.261944 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:51.262059 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":450,"threads_started":1}
W20260929 02:53:51.262578 22038 catalog_manager.cc:4584] TS a75aa73e1ab847d1943d881186533cc1 (127.4.197.129:44971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15680597349c4f3a85da45d261d9cc54: Network error: Client connection negotiation failed: client connection to 127.4.197.129:44971: connect: Connection refused (error 111)
Sep 29 02:53:51 dist-test-slave-t6c1 krb5kdc[22007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:53:51.277581 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.261427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.4.197.130:36557)
0929 02:53:51.261605 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:51.261611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:51.261885 (+ 274us) server_negotiation.cc:407] Connection header received
0929 02:53:51.261958 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:51.261963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:51.262055 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:51.262156 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:51.263946 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:51.264663 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:51.266529 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:51.266975 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:51.267078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:51.271894 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:51.271909 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:51.271924 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:51.271960 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:51.274644 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:51.275473 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:51.275481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:51.275483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:51.275619 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:51.275997 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:51.276001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:51.276003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:51.276401 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:51.276674 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:51.277030 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 02:53:51.277268 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:53:51.278271 22370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:51.316128 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.315543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:44971 (local address 127.0.0.1:47618)
0929 02:53:51.315820 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:51.315944 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 02:53:51.317373 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.316923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:44971 (local address 127.0.0.1:47620)
0929 02:53:51.317098 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:51.317202 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:53:51.369594 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 22247
I20260929 02:53:51.378105 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.377570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:44971 (local address 127.0.0.1:47628)
0929 02:53:51.377815 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:51.377925 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 02:53:51.379407 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.378952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:44971 (local address 127.0.0.1:47642)
0929 02:53:51.379101 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:51.379200 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 02:53:51.384447 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 22380
Sep 29 02:53:51 dist-test-slave-t6c1 krb5kdc[22007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:53:51.401911 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.385347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.4.197.131:41601)
0929 02:53:51.385573 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:51.385579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:51.385596 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:53:51.385649 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:51.385653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:51.385712 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:51.385802 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:51.387153 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:51.387964 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:51.389920 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:51.390372 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:51.390468 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:51.396048 (+ 5580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:51.396062 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:51.396076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:51.396115 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:51.399151 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:51.399788 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:51.399795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:51.399797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:51.399946 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:51.400296 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:51.400301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:51.400303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:51.400717 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:51.401032 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:51.401326 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 02:53:51.401558 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:53:51.402710 22503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:51.460114 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.459535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:44971 (local address 127.0.0.1:47644)
0929 02:53:51.459820 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:51.459939 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 02:53:51.461220 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:51.460764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:44971 (local address 127.0.0.1:47654)
0929 02:53:51.460963 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:51.461048 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 02:53:51.527647 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 22380
I20260929 02:53:51.543440 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 22020
I20260929 02:53:51.544524 22082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:51.654524 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 22020
2026-09-29T02:53:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4471 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T02:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:53:51Z Disabled control of system clock
I20260929 02:53:51.720368 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:40691
--webserver_interface=127.4.197.190
--webserver_port=0
--builtin_ntp_servers=127.4.197.148:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:40691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:53:52.041502 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:52.041803 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:52.041925 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:52.049409 22563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:53:52.049543 22563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:52.049609 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:52.049688 22563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:53:52.049746 22563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:53:52.059247 22563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:40691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:40691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:52.060706 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:52.062795 22563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:52.074736 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:52.074692 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:52.074774 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:52.075614 22563 server_base.cc:1034] running on GCE node
I20260929 02:53:52.076395 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:52.077740 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:52.078948 22563 hybrid_clock.cc:648] HybridClock initialized: now 1790650432078867 us; error 47 us; skew 500 ppm
I20260929 02:53:52.079303 22563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:53:52.085006 22563 webserver.cc:460] Webserver started at http://127.4.197.190:39491/ using document root <none> and password file <none>
I20260929 02:53:52.085695 22563 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:52.085788 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:52.086088 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:52.089004 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "0255e543becc4bb4a7d75ab67795fcff"
format_stamp: "Formatted at 2026-09-29 02:53:52 on dist-test-slave-t6c1"
server_key: "5d02c3cef43fc8c1ac9e42eb3a5d4329"
server_key_iv: "7e226d4da9948051034432f6ba3be2fc"
server_key_version: "encryptionkey@0"
I20260929 02:53:52.089785 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "0255e543becc4bb4a7d75ab67795fcff"
format_stamp: "Formatted at 2026-09-29 02:53:52 on dist-test-slave-t6c1"
server_key: "5d02c3cef43fc8c1ac9e42eb3a5d4329"
server_key_iv: "7e226d4da9948051034432f6ba3be2fc"
server_key_version: "encryptionkey@0"
I20260929 02:53:52.095198 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:53:52.098757 22577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:52.100127 22563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:53:52.100289 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "0255e543becc4bb4a7d75ab67795fcff"
format_stamp: "Formatted at 2026-09-29 02:53:52 on dist-test-slave-t6c1"
server_key: "5d02c3cef43fc8c1ac9e42eb3a5d4329"
server_key_iv: "7e226d4da9948051034432f6ba3be2fc"
server_key_version: "encryptionkey@0"
I20260929 02:53:52.100500 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:52.111286 22563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:52.112044 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:52.112301 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:52.128226 22563 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:40691
I20260929 02:53:52.128238 22628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:40691 every 8 connection(s)
I20260929 02:53:52.129501 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:53:52.131697 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 22563
I20260929 02:53:52.131839 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:53:52.132270 4886 external_mini_cluster.cc:1428] Setting key 7728e9e4de15e2eb86b468c110776903
I20260929 02:53:52.133512 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:52.145730 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:52.134118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.4.197.190:40691)
0929 02:53:52.135095 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:52.135106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:52.135149 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:53:52.136615 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:52.136635 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:52.137187 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:52.137543 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:52.138165 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:52.139583 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:52.141441 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:52.142083 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:52.142205 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:52.143495 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:52.143503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:52.143519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:52.144086 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:52.144161 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:52.144181 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:52.144269 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:52.144495 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 02:53:52.144805 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":330,"threads_started":1}
I20260929 02:53:52.146598 22629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff: Bootstrap starting.
I20260929 02:53:52.150031 22629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:52.151014 22629 log.cc:824] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:52.153730 22629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff: No bootstrap required, opened a new log
I20260929 02:53:52.160916 22629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255e543becc4bb4a7d75ab67795fcff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40691 } }
I20260929 02:53:52.161265 22629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:52.161355 22629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0255e543becc4bb4a7d75ab67795fcff, State: Initialized, Role: FOLLOWER
I20260929 02:53:52.161887 22629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255e543becc4bb4a7d75ab67795fcff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40691 } }
I20260929 02:53:52.162104 22629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:53:52.162204 22629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:53:52.162325 22629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:52.164701 22629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255e543becc4bb4a7d75ab67795fcff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40691 } }
I20260929 02:53:52.165144 22629 leader_election.cc:302] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0255e543becc4bb4a7d75ab67795fcff; no voters:
I20260929 02:53:52.165686 22629 leader_election.cc:288] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:53:52.165804 22634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:53:52.166487 22634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [term 1 LEADER]: Becoming Leader. State: Replica: 0255e543becc4bb4a7d75ab67795fcff, State: Running, Role: LEADER
I20260929 02:53:52.167047 22634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255e543becc4bb4a7d75ab67795fcff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40691 } }
I20260929 02:53:52.167748 22629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:53:52.169813 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0255e543becc4bb4a7d75ab67795fcff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255e543becc4bb4a7d75ab67795fcff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40691 } } }
I20260929 02:53:52.170315 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:52.169836 22636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0255e543becc4bb4a7d75ab67795fcff. Latest consensus state: current_term: 1 leader_uuid: "0255e543becc4bb4a7d75ab67795fcff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0255e543becc4bb4a7d75ab67795fcff" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 40691 } } }
I20260929 02:53:52.170560 22636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:52.172969 22640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:53:52.177309 22640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:53:52.185786 22640 catalog_manager.cc:1347] Generated new cluster ID: d1a27fa629d045378cb441671203c22a
I20260929 02:53:52.185874 22640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:53:52.197467 22640 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:53:52.198556 22640 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:53:52.207840 22640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff: Generated new TSK 0
I20260929 02:53:52.208705 22640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:53:52.217087 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40691
--builtin_ntp_servers=127.4.197.148:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:53:52.564366 22653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:52.564682 22653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:52.564771 22653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:52.572238 22653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:52.572366 22653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:52.572489 22653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:53:52.581648 22653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:52.582978 22653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:52.584741 22653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:52.597674 22661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:52.597708 22658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:52.597668 22659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:52.598448 22653 server_base.cc:1034] running on GCE node
I20260929 02:53:52.599136 22653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:52.600104 22653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:52.601308 22653 hybrid_clock.cc:648] HybridClock initialized: now 1790650432601248 us; error 53 us; skew 500 ppm
I20260929 02:53:52.601624 22653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:53:52.604022 22653 webserver.cc:460] Webserver started at http://127.4.197.129:32915/ using document root <none> and password file <none>
I20260929 02:53:52.604815 22653 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:52.604938 22653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:52.605260 22653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:52.608549 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "d7225881f1b340b080a7a122d68ccf07"
format_stamp: "Formatted at 2026-09-29 02:53:52 on dist-test-slave-t6c1"
server_key: "aab98733cc9b392a3383caa482e1bccb"
server_key_iv: "c47bb2ff7375d52b99392378c1f6048d"
server_key_version: "encryptionkey@0"
I20260929 02:53:52.609330 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "d7225881f1b340b080a7a122d68ccf07"
format_stamp: "Formatted at 2026-09-29 02:53:52 on dist-test-slave-t6c1"
server_key: "aab98733cc9b392a3383caa482e1bccb"
server_key_iv: "c47bb2ff7375d52b99392378c1f6048d"
server_key_version: "encryptionkey@0"
I20260929 02:53:52.614387 22653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 02:53:52.617914 22667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:52.619280 22653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:53:52.619446 22653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "d7225881f1b340b080a7a122d68ccf07"
format_stamp: "Formatted at 2026-09-29 02:53:52 on dist-test-slave-t6c1"
server_key: "aab98733cc9b392a3383caa482e1bccb"
server_key_iv: "c47bb2ff7375d52b99392378c1f6048d"
server_key_version: "encryptionkey@0"
I20260929 02:53:52.619637 22653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:52.638617 22653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:52.639635 22653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:52.639945 22653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:52.640995 22653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:52.642704 22653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:52.642802 22653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:52.642910 22653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:52.643034 22653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 02:53:52.673223 22653 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:40999
I20260929 02:53:52.673271 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:40999 every 8 connection(s)
I20260929 02:53:52.674685 22653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:53:52.683957 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 22653
I20260929 02:53:52.684125 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:53:52.684607 4886 external_mini_cluster.cc:1428] Setting key 8093ad19e6b1130019a9e08ea8cb96e1
I20260929 02:53:52.687984 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40691
--builtin_ntp_servers=127.4.197.148:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:53:52.688239 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:52.677400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:44855 (local address 127.4.197.190:40691)
0929 02:53:52.677826 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:52.677830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:52.679392 (+ 1562us) server_negotiation.cc:407] Connection header received
0929 02:53:52.681249 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:52.681253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:52.681327 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:52.681432 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:52.682539 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:52.683316 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:52.685018 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:52.685539 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:52.685700 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:52.687360 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:52.687366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:52.687371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:52.687399 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:52.687470 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:52.687492 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:52.687587 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:52.687768 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 02:53:52.687886 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":206,"threads_started":1}
I20260929 02:53:52.689249 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:52.678097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:40691 (local address 127.4.197.129:44855)
0929 02:53:52.679209 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:52.679259 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:52.680982 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:52.681651 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:52.681661 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:52.681769 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:52.682410 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:52.682427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:52.683446 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:52.683450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:52.684872 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:52.684885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:52.685790 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:52.685799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:52.686119 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:52.686909 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:53:52.686912 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:52.687010 (+ 98us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:53:52.687133 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:52.687579 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:52.687586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:52.687591 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:53:52.687732 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 02:53:52.688084 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":307,"threads_started":1}
I20260929 02:53:52.690905 22780 heartbeater.cc:347] Connected to a master server at 127.4.197.190:40691
I20260929 02:53:52.691269 22780 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:52.692234 22780 heartbeater.cc:511] Master 127.4.197.190:40691 requested a full tablet report, sending...
I20260929 02:53:52.694598 22594 ts_manager.cc:194] Registered new tserver with Master: d7225881f1b340b080a7a122d68ccf07 (127.4.197.129:40999)
I20260929 02:53:52.696403 22594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.129:44855
W20260929 02:53:53.010445 22784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:53.010691 22784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:53.010818 22784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:53.017721 22784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:53.017802 22784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:53.017943 22784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:53:53.026594 22784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:53.027832 22784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:53.029595 22784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:53.042706 22789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:53.042706 22792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:53.042665 22790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:53.043431 22784 server_base.cc:1034] running on GCE node
I20260929 02:53:53.044077 22784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:53.045053 22784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:53.046252 22784 hybrid_clock.cc:648] HybridClock initialized: now 1790650433046187 us; error 60 us; skew 500 ppm
I20260929 02:53:53.046567 22784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:53:53.048770 22784 webserver.cc:460] Webserver started at http://127.4.197.130:37895/ using document root <none> and password file <none>
I20260929 02:53:53.049479 22784 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:53.049577 22784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:53.049881 22784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:53.052857 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "54b75531a4ad4deeaf18779f6beb8eae"
format_stamp: "Formatted at 2026-09-29 02:53:53 on dist-test-slave-t6c1"
server_key: "91adabe5af01a15744fcb9c8cb4de1e5"
server_key_iv: "e68ff3b644ef58c98d7d5a6976d3ad16"
server_key_version: "encryptionkey@0"
I20260929 02:53:53.053725 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "54b75531a4ad4deeaf18779f6beb8eae"
format_stamp: "Formatted at 2026-09-29 02:53:53 on dist-test-slave-t6c1"
server_key: "91adabe5af01a15744fcb9c8cb4de1e5"
server_key_iv: "e68ff3b644ef58c98d7d5a6976d3ad16"
server_key_version: "encryptionkey@0"
I20260929 02:53:53.058787 22784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:53:53.062105 22798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:53.063462 22784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:53:53.063606 22784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "54b75531a4ad4deeaf18779f6beb8eae"
format_stamp: "Formatted at 2026-09-29 02:53:53 on dist-test-slave-t6c1"
server_key: "91adabe5af01a15744fcb9c8cb4de1e5"
server_key_iv: "e68ff3b644ef58c98d7d5a6976d3ad16"
server_key_version: "encryptionkey@0"
I20260929 02:53:53.063795 22784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:53.106707 22784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:53.107561 22784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:53.107832 22784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:53.108767 22784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:53.110148 22784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:53.110239 22784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:53.110342 22784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:53.110409 22784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:53.141059 22784 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:42295
I20260929 02:53:53.141099 22910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:42295 every 8 connection(s)
I20260929 02:53:53.142309 22784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:53:53.142609 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 22784
I20260929 02:53:53.142784 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:53:53.143463 4886 external_mini_cluster.cc:1428] Setting key bb8781cf852b8b7d6ed693e2e167cbcf
I20260929 02:53:53.148223 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40691
--builtin_ntp_servers=127.4.197.148:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:53:53.156997 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.146037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:55959 (local address 127.4.197.190:40691)
0929 02:53:53.146191 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:53.146196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:53.148088 (+ 1892us) server_negotiation.cc:407] Connection header received
0929 02:53:53.150021 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:53.150028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:53.150110 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:53.150221 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:53.151479 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.152464 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.154080 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.154626 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.154747 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.156251 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:53.156257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:53.156260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:53.156283 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:53.156319 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:53.156331 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:53.156419 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:53.156659 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 02:53:53.156748 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:53:53.158432 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.146738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:40691 (local address 127.4.197.130:55959)
0929 02:53:53.147894 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:53.147951 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:53.149722 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:53.150500 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:53.150510 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:53.150612 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:53.151298 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.151318 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.152625 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.152629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.153934 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.153947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.154829 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.154833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.155103 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.155851 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:53:53.155855 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:53.155920 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:53:53.156035 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:53.156440 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:53.156452 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:53.156461 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:53:53.156649 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 02:53:53.157074 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":368,"threads_started":1}
I20260929 02:53:53.159901 22911 heartbeater.cc:347] Connected to a master server at 127.4.197.190:40691
I20260929 02:53:53.160200 22911 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:53.160962 22911 heartbeater.cc:511] Master 127.4.197.190:40691 requested a full tablet report, sending...
I20260929 02:53:53.162346 22594 ts_manager.cc:194] Registered new tserver with Master: 54b75531a4ad4deeaf18779f6beb8eae (127.4.197.130:42295)
I20260929 02:53:53.163223 22594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.130:55959
W20260929 02:53:53.469748 22914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:53.470062 22914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:53.470196 22914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:53.477865 22914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:53.478040 22914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:53.478196 22914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:53:53.488224 22914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:53.489570 22914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:53.491452 22914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:53.504923 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:53.504926 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:53.504868 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:53.506152 22914 server_base.cc:1034] running on GCE node
I20260929 02:53:53.506839 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:53.507947 22914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:53.509181 22914 hybrid_clock.cc:648] HybridClock initialized: now 1790650433509124 us; error 57 us; skew 500 ppm
I20260929 02:53:53.509536 22914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:53:53.512171 22914 webserver.cc:460] Webserver started at http://127.4.197.131:34609/ using document root <none> and password file <none>
I20260929 02:53:53.512921 22914 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:53.513032 22914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:53.513348 22914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:53.516357 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "cef1ee2949c94ee6bd01f57c1630b680"
format_stamp: "Formatted at 2026-09-29 02:53:53 on dist-test-slave-t6c1"
server_key: "1669a7abf0eb7a75b81adb97e959e1ff"
server_key_iv: "1b0fabb44ef9662bde19ea078cb131b3"
server_key_version: "encryptionkey@0"
I20260929 02:53:53.517151 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "cef1ee2949c94ee6bd01f57c1630b680"
format_stamp: "Formatted at 2026-09-29 02:53:53 on dist-test-slave-t6c1"
server_key: "1669a7abf0eb7a75b81adb97e959e1ff"
server_key_iv: "1b0fabb44ef9662bde19ea078cb131b3"
server_key_version: "encryptionkey@0"
I20260929 02:53:53.522632 22914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:53:53.526782 22928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:53.528573 22914 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 02:53:53.528780 22914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "cef1ee2949c94ee6bd01f57c1630b680"
format_stamp: "Formatted at 2026-09-29 02:53:53 on dist-test-slave-t6c1"
server_key: "1669a7abf0eb7a75b81adb97e959e1ff"
server_key_iv: "1b0fabb44ef9662bde19ea078cb131b3"
server_key_version: "encryptionkey@0"
I20260929 02:53:53.529023 22914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:53.557112 22914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:53.558013 22914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:53.558308 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:53.559324 22914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:53.560815 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:53.560927 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:53.561045 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:53.561116 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:53.591755 22914 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:44841
I20260929 02:53:53.591802 23040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:44841 every 8 connection(s)
I20260929 02:53:53.593027 22914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:53:53.603021 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 22914
I20260929 02:53:53.603173 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:53:53.603581 4886 external_mini_cluster.cc:1428] Setting key 3c438d81dac1505f9230f1bdc373cbd5
I20260929 02:53:53.605924 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.595643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:34953 (local address 127.4.197.190:40691)
0929 02:53:53.595814 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:53.595819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:53.597235 (+ 1416us) server_negotiation.cc:407] Connection header received
0929 02:53:53.599203 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:53.599208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:53.599286 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:53.599369 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:53.600500 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.601265 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.602914 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.603476 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.603604 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.605077 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:53.605083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:53.605088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:53.605108 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:53.605142 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:53.605153 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:53.605226 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:53.605478 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 02:53:53.605578 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:53:53.607059 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.596148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:40691 (local address 127.4.197.131:34953)
0929 02:53:53.597079 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:53.597117 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:53.598869 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:53.599564 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:53.599573 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:53.599666 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:53.600312 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.600330 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.601450 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.601455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.602740 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.602753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.603706 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.603713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.603964 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.604727 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:53:53.604730 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:53.604789 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:53:53.604893 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:53.605236 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:53.605246 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:53.605254 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:53:53.605453 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 02:53:53.605844 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":274,"threads_started":1}
I20260929 02:53:53.608306 23041 heartbeater.cc:347] Connected to a master server at 127.4.197.190:40691
I20260929 02:53:53.608582 23041 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:53.609261 23041 heartbeater.cc:511] Master 127.4.197.190:40691 requested a full tablet report, sending...
I20260929 02:53:53.610535 22594 ts_manager.cc:194] Registered new tserver with Master: cef1ee2949c94ee6bd01f57c1630b680 (127.4.197.131:44841)
I20260929 02:53:53.611348 22594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.131:34953
I20260929 02:53:53.618955 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:53:53.629212 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.622959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.4.197.190:40691)
0929 02:53:53.623118 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:53.623122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:53.623611 (+ 489us) server_negotiation.cc:407] Connection header received
0929 02:53:53.623773 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:53.623777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:53.623842 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:53.623974 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:53.624559 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.625266 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.626698 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.627183 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.627291 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.628553 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:53.628558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:53.628562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:53.628584 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:53.628621 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:53.628633 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:53.628713 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:53.628888 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 02:53:53.628976 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:53:53.633414 22594 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:53:53.640342 22594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:53:53.661417 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.653122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.4.197.129:40999)
0929 02:53:53.654236 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:53.654254 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:53.654284 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:53:53.654389 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:53.654398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:53.654676 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:53.655579 (+ 903us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:53.655823 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.657233 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.659113 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.660248 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.660372 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.660632 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:53.660904 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 02:53:53.661070 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":835,"threads_started":1}
I20260929 02:53:53.661695 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.652961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40999 (local address 127.0.0.1:47532)
0929 02:53:53.653581 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:53.653617 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:53.653764 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:53.654961 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:53.654973 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:53.655004 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:53.655388 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.655399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.657350 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.657355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.658903 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.658940 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.660437 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.660443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.660659 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.660725 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 02:53:53.660840 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 02:53:53.661046 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":3,"thread_start_us":295,"threads_started":1}
I20260929 02:53:53.663902 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.654133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:44841 (local address 127.0.0.1:33252)
0929 02:53:53.654649 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:53.654666 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:53.654776 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:53.656473 (+ 1697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:53.656479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:53.656504 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:53.656894 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.656908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.659105 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.659110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.661349 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.661368 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.663322 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.663327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.663462 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.663487 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:53:53.663547 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 02:53:53.663646 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":31,"thread_start_us":265,"threads_started":1}
I20260929 02:53:53.664167 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.654542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.4.197.131:44841)
0929 02:53:53.655884 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:53.655893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:53.655926 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:53:53.656016 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:53.656024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:53.656272 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:53.657084 (+ 812us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:53.657358 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.658949 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.661737 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.663157 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.663292 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.663557 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:53.663632 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 02:53:53.663804 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":1041,"threads_started":1}
I20260929 02:53:53.665467 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.654693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:42295 (local address 127.0.0.1:35190)
0929 02:53:53.655242 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:53.655262 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:53.655377 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:53.656747 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:53.656751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:53.656781 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:53.657204 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.657216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.659403 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.659409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.661436 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.661447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.664676 (+ 3229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.664682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.664854 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.664885 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:53:53.664968 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 02:53:53.665101 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":23,"thread_start_us":799,"threads_started":1}
I20260929 02:53:53.665524 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.655322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35190 (local address 127.4.197.130:42295)
0929 02:53:53.656164 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:53.656172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:53.656200 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:53:53.656294 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:53.656301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:53.656526 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:53.656749 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:53.657391 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.659240 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.661642 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.664503 (+ 2861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.664621 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.664863 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:53.665058 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 02:53:53.665225 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":1159,"threads_started":1}
I20260929 02:53:53.668067 22976 tablet_service.cc:1467] Processing CreateTablet for tablet 96f3aea7a72f4391a725f97f0f3b1059 (DEFAULT_TABLE table=test-table [id=b143b2b5131d4805b2a539ec269fc8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:53.669663 22976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f3aea7a72f4391a725f97f0f3b1059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:53.669390 22846 tablet_service.cc:1467] Processing CreateTablet for tablet 96f3aea7a72f4391a725f97f0f3b1059 (DEFAULT_TABLE table=test-table [id=b143b2b5131d4805b2a539ec269fc8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:53.670746 22846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f3aea7a72f4391a725f97f0f3b1059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:53.666139 22715 tablet_service.cc:1467] Processing CreateTablet for tablet 96f3aea7a72f4391a725f97f0f3b1059 (DEFAULT_TABLE table=test-table [id=b143b2b5131d4805b2a539ec269fc8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:53.677105 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f3aea7a72f4391a725f97f0f3b1059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:53.684532 23059 tablet_bootstrap.cc:492] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: Bootstrap starting.
I20260929 02:53:53.686722 23060 tablet_bootstrap.cc:492] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: Bootstrap starting.
I20260929 02:53:53.688370 23059 tablet_bootstrap.cc:654] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:53.689603 23060 tablet_bootstrap.cc:654] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:53.689654 23059 log.cc:824] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:53.690959 23060 log.cc:824] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:53.691958 23062 tablet_bootstrap.cc:492] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07: Bootstrap starting.
I20260929 02:53:53.693887 23059 tablet_bootstrap.cc:492] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: No bootstrap required, opened a new log
I20260929 02:53:53.694038 23060 tablet_bootstrap.cc:492] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: No bootstrap required, opened a new log
I20260929 02:53:53.694180 23059 ts_tablet_manager.cc:1397] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260929 02:53:53.694289 23060 ts_tablet_manager.cc:1397] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:53:53.696306 23062 tablet_bootstrap.cc:654] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:53.697489 23062 log.cc:824] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:53.699440 22780 heartbeater.cc:503] Master 127.4.197.190:40691 was elected leader, sending a full tablet report...
I20260929 02:53:53.700578 23062 tablet_bootstrap.cc:492] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07: No bootstrap required, opened a new log
I20260929 02:53:53.700907 23062 ts_tablet_manager.cc:1397] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:53:53.703949 23060 raft_consensus.cc:348] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225881f1b340b080a7a122d68ccf07" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40999 } } peers { permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 } } peers { permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 } }
I20260929 02:53:53.704525 23060 raft_consensus.cc:374] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:53.704630 23060 raft_consensus.cc:729] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b75531a4ad4deeaf18779f6beb8eae, State: Initialized, Role: FOLLOWER
I20260929 02:53:53.705019 23059 raft_consensus.cc:348] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225881f1b340b080a7a122d68ccf07" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40999 } } peers { permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 } } peers { permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 } }
I20260929 02:53:53.705349 23059 raft_consensus.cc:374] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:53.705446 23059 raft_consensus.cc:729] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef1ee2949c94ee6bd01f57c1630b680, State: Initialized, Role: FOLLOWER
I20260929 02:53:53.705299 23060 consensus_queue.cc:260] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225881f1b340b080a7a122d68ccf07" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40999 } } peers { permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 } } peers { permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 } }
I20260929 02:53:53.705945 23059 consensus_queue.cc:260] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225881f1b340b080a7a122d68ccf07" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40999 } } peers { permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 } } peers { permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 } }
I20260929 02:53:53.706964 23041 heartbeater.cc:503] Master 127.4.197.190:40691 was elected leader, sending a full tablet report...
I20260929 02:53:53.707476 23059 ts_tablet_manager.cc:1428] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260929 02:53:53.708731 22911 heartbeater.cc:503] Master 127.4.197.190:40691 was elected leader, sending a full tablet report...
I20260929 02:53:53.709390 23060 ts_tablet_manager.cc:1428] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260929 02:53:53.710646 23062 raft_consensus.cc:348] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225881f1b340b080a7a122d68ccf07" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40999 } } peers { permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 } } peers { permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 } }
I20260929 02:53:53.711093 23062 raft_consensus.cc:374] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:53.711184 23062 raft_consensus.cc:729] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7225881f1b340b080a7a122d68ccf07, State: Initialized, Role: FOLLOWER
I20260929 02:53:53.711807 23062 consensus_queue.cc:260] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225881f1b340b080a7a122d68ccf07" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40999 } } peers { permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 } } peers { permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 } }
I20260929 02:53:53.713404 23062 ts_tablet_manager.cc:1428] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07: Time spent starting tablet: real 0.012s user 0.003s sys 0.008s
I20260929 02:53:53.762271 23067 raft_consensus.cc:482] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:53:53.762566 23067 raft_consensus.cc:504] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225881f1b340b080a7a122d68ccf07" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40999 } } peers { permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 } } peers { permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 } }
I20260929 02:53:53.764288 23067 leader_election.cc:288] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cef1ee2949c94ee6bd01f57c1630b680 (127.4.197.131:44841), 54b75531a4ad4deeaf18779f6beb8eae (127.4.197.130:42295)
I20260929 02:53:53.771313 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.764706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:59619 (local address 127.4.197.131:44841)
0929 02:53:53.764937 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:53.764944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:53.765219 (+ 275us) server_negotiation.cc:407] Connection header received
0929 02:53:53.765481 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:53.765487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:53.765568 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:53.765636 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:53.766379 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.767500 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.769584 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.770374 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.770556 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.770637 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:53.770902 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 02:53:53.771072 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:53:53.771425 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.764521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:44841 (local address 127.4.197.129:59619)
0929 02:53:53.765073 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:53.765092 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:53.765289 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:53.765827 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:53.765831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:53.765852 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:53.766221 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.766237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.767670 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.767675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.769382 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.769403 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.770556 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.770563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.770745 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.770770 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:53:53.770854 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 02:53:53.771033 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":4,"thread_start_us":263,"threads_started":1}
I20260929 02:53:53.771466 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.765146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:42295 (local address 127.4.197.129:46765)
0929 02:53:53.765682 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:53.765698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:53.765822 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:53.766111 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:53.766116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:53.766141 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:53.766482 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.766492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.767850 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.767854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.769494 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:53.769508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.770685 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.770694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:53.770852 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.770876 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:53:53.770995 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 02:53:53.771128 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":241,"threads_started":1}
I20260929 02:53:53.771924 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.765412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:46765 (local address 127.4.197.130:42295)
0929 02:53:53.765605 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:53.765609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:53.765763 (+ 154us) server_negotiation.cc:407] Connection header received
0929 02:53:53.765934 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:53.765938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:53.765997 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:53.766069 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:53.766613 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.767681 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.769681 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.770493 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.770653 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.770723 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:53.771027 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 02:53:53.771160 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:53:53.772518 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f3aea7a72f4391a725f97f0f3b1059" candidate_uuid: "d7225881f1b340b080a7a122d68ccf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef1ee2949c94ee6bd01f57c1630b680" is_pre_election: true
I20260929 02:53:53.772822 22866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f3aea7a72f4391a725f97f0f3b1059" candidate_uuid: "d7225881f1b340b080a7a122d68ccf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b75531a4ad4deeaf18779f6beb8eae" is_pre_election: true
I20260929 02:53:53.773203 22996 raft_consensus.cc:2454] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7225881f1b340b080a7a122d68ccf07 in term 0.
I20260929 02:53:53.773388 22866 raft_consensus.cc:2454] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7225881f1b340b080a7a122d68ccf07 in term 0.
I20260929 02:53:53.773973 22671 leader_election.cc:302] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cef1ee2949c94ee6bd01f57c1630b680, d7225881f1b340b080a7a122d68ccf07; no voters:
I20260929 02:53:53.774371 23067 raft_consensus.cc:2781] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:53:53.774506 23067 raft_consensus.cc:482] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:53:53.774622 23067 raft_consensus.cc:3037] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:53.776909 23067 raft_consensus.cc:504] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225881f1b340b080a7a122d68ccf07" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40999 } } peers { permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 } } peers { permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 } }
I20260929 02:53:53.777534 23067 leader_election.cc:288] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [CANDIDATE]: Term 1 election: Requested vote from peers cef1ee2949c94ee6bd01f57c1630b680 (127.4.197.131:44841), 54b75531a4ad4deeaf18779f6beb8eae (127.4.197.130:42295)
I20260929 02:53:53.778124 22996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f3aea7a72f4391a725f97f0f3b1059" candidate_uuid: "d7225881f1b340b080a7a122d68ccf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef1ee2949c94ee6bd01f57c1630b680"
I20260929 02:53:53.778175 22866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f3aea7a72f4391a725f97f0f3b1059" candidate_uuid: "d7225881f1b340b080a7a122d68ccf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b75531a4ad4deeaf18779f6beb8eae"
I20260929 02:53:53.778354 22996 raft_consensus.cc:3037] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:53.778355 22866 raft_consensus.cc:3037] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:53.780781 22866 raft_consensus.cc:2454] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7225881f1b340b080a7a122d68ccf07 in term 1.
I20260929 02:53:53.780781 22996 raft_consensus.cc:2454] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7225881f1b340b080a7a122d68ccf07 in term 1.
I20260929 02:53:53.781250 22671 leader_election.cc:302] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54b75531a4ad4deeaf18779f6beb8eae, d7225881f1b340b080a7a122d68ccf07; no voters:
I20260929 02:53:53.781649 23067 raft_consensus.cc:2781] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:53:53.782102 23067 raft_consensus.cc:686] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 1 LEADER]: Becoming Leader. State: Replica: d7225881f1b340b080a7a122d68ccf07, State: Running, Role: LEADER
I20260929 02:53:53.782613 23067 consensus_queue.cc:237] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225881f1b340b080a7a122d68ccf07" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40999 } } peers { permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 } } peers { permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 } }
I20260929 02:53:53.787453 22592 catalog_manager.cc:5495] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7225881f1b340b080a7a122d68ccf07 (127.4.197.129). New cstate: current_term: 1 leader_uuid: "d7225881f1b340b080a7a122d68ccf07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225881f1b340b080a7a122d68ccf07" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 40999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 } health_report { overall_health: UNKNOWN } } }
I20260929 02:53:53.826174 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.820879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.4.197.129:40999)
0929 02:53:53.821040 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:53.821045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:53.821062 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:53:53.821149 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:53.821153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:53.821247 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:53.821413 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:53:53.821713 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.822413 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.824005 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.824431 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.824545 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.825071 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:53:53.825602 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:53:53.825740 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:53.825824 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 02:53:53.825906 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:53:53.835177 22866 raft_consensus.cc:1261] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae [term 1 FOLLOWER]: Refusing update from remote peer d7225881f1b340b080a7a122d68ccf07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:53:53.835177 22996 raft_consensus.cc:1261] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680 [term 1 FOLLOWER]: Refusing update from remote peer d7225881f1b340b080a7a122d68ccf07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:53:53.836253 23070 consensus_queue.cc:1035] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef1ee2949c94ee6bd01f57c1630b680" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:53:53.836635 23067 consensus_queue.cc:1035] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54b75531a4ad4deeaf18779f6beb8eae" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 42295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:53:53.844738 23042 tablet.cc:2369] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:53:53.861037 22593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:53:53.861284 22593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:53:53.864743 22593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0255e543becc4bb4a7d75ab67795fcff: Sending DeleteTablet for 3 replicas of tablet 96f3aea7a72f4391a725f97f0f3b1059
I20260929 02:53:53.865720 22715 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f3aea7a72f4391a725f97f0f3b1059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:53 UTC) from {username='slave'} at 127.0.0.1:47532
I20260929 02:53:53.865983 22846 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f3aea7a72f4391a725f97f0f3b1059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:53 UTC) from {username='slave'} at 127.0.0.1:35190
I20260929 02:53:53.866035 22976 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f3aea7a72f4391a725f97f0f3b1059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:53 UTC) from {username='slave'} at 127.0.0.1:33252
I20260929 02:53:53.867743 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 22653
I20260929 02:53:53.877441 23085 tablet_replica.cc:330] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: stopping tablet replica
I20260929 02:53:53.877985 23085 raft_consensus.cc:2229] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:53:53.878561 23085 raft_consensus.cc:2258] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:53.879516 23084 tablet_replica.cc:330] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: stopping tablet replica
I20260929 02:53:53.879972 23084 raft_consensus.cc:2229] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:53:53.880487 23084 raft_consensus.cc:2258] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:53.881857 23083 tablet_replica.cc:330] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07: stopping tablet replica
I20260929 02:53:53.882273 23083 raft_consensus.cc:2229] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:53:53.882730 23085 ts_tablet_manager.cc:1905] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:53.882956 23083 raft_consensus.cc:2258] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:53.884760 23084 ts_tablet_manager.cc:1905] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:53.889556 23083 ts_tablet_manager.cc:1905] T 96f3aea7a72f4391a725f97f0f3b1059 P d7225881f1b340b080a7a122d68ccf07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:53.890100 23085 ts_tablet_manager.cc:1918] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:53.890252 23085 log.cc:1196] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/96f3aea7a72f4391a725f97f0f3b1059
I20260929 02:53:53.890715 23085 ts_tablet_manager.cc:1939] T 96f3aea7a72f4391a725f97f0f3b1059 P cef1ee2949c94ee6bd01f57c1630b680: Deleting consensus metadata
I20260929 02:53:53.890435 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:53.868762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.4.197.129:40999)
0929 02:53:53.868917 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:53.868924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:53.879874 (+ 10950us) server_negotiation.cc:407] Connection header received
0929 02:53:53.880242 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:53.880249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:53.880354 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:53.880438 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:53.881202 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.882451 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.885029 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:53.885675 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:53.885792 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:53.887413 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:53.887425 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:53.887442 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:53.888271 (+ 829us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:53.888339 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:53.888365 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:53.888442 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:53.889951 (+ 1509us) server_negotiation.cc:299] Negotiation successful
0929 02:53:53.890069 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:53:53.891774 23084 ts_tablet_manager.cc:1918] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:53.891832 22775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:53.891934 23084 log.cc:1196] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/96f3aea7a72f4391a725f97f0f3b1059
I20260929 02:53:53.892428 23084 ts_tablet_manager.cc:1939] T 96f3aea7a72f4391a725f97f0f3b1059 P 54b75531a4ad4deeaf18779f6beb8eae: Deleting consensus metadata
I20260929 02:53:53.892629 22581 catalog_manager.cc:4841] TS cef1ee2949c94ee6bd01f57c1630b680 (127.4.197.131:44841): tablet 96f3aea7a72f4391a725f97f0f3b1059 (table test-table [id=b143b2b5131d4805b2a539ec269fc8a0]) successfully deleted
I20260929 02:53:53.893957 22581 catalog_manager.cc:4841] TS 54b75531a4ad4deeaf18779f6beb8eae (127.4.197.130:42295): tablet 96f3aea7a72f4391a725f97f0f3b1059 (table test-table [id=b143b2b5131d4805b2a539ec269fc8a0]) successfully deleted
I20260929 02:53:53.990263 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 22653
I20260929 02:53:54.004874 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 22784
W20260929 02:53:54.004920 22581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.197.129:40999 (error 108)
I20260929 02:53:54.006737 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:54.006269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40999 (local address 127.0.0.1:47550)
0929 02:53:54.006427 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:54.006549 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:53:54.006975 22581 catalog_manager.cc:4584] TS d7225881f1b340b080a7a122d68ccf07 (127.4.197.129:40999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96f3aea7a72f4391a725f97f0f3b1059: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40999: connect: Connection refused (error 111)
I20260929 02:53:54.011266 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:54.005618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.4.197.130:42295)
0929 02:53:54.005791 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:54.005795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:54.005905 (+ 110us) server_negotiation.cc:407] Connection header received
0929 02:53:54.006106 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:54.006112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:54.006216 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:54.006319 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:54.006715 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:54.007590 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:54.008740 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:54.009114 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:54.009196 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:54.010079 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:54.010087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:54.010100 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:54.010657 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:54.010699 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:54.010717 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:54.010798 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:54.010874 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 02:53:54.010986 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:53:54.011811 22906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:54.060576 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:54.060008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40999 (local address 127.0.0.1:47562)
0929 02:53:54.060253 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:54.060375 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:53:54.061654 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:54.061284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40999 (local address 127.0.0.1:47566)
0929 02:53:54.061422 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:54.061509 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 02:53:54.098752 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 22784
I20260929 02:53:54.111799 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 22914
I20260929 02:53:54.117970 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:54.112464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.4.197.131:44841)
0929 02:53:54.112615 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:54.112619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:54.112637 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:53:54.112792 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:54.112796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:54.112878 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:54.112960 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:54.113372 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:54.114119 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:54.115394 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:54.115777 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:54.115868 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:54.116789 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:54.116797 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:54.116812 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:54.117377 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:54.117427 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:54.117450 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:54.117557 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:54.117608 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 02:53:54.117715 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:53:54.118559 23036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:54.122838 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:54.122363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40999 (local address 127.0.0.1:47582)
0929 02:53:54.122575 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:54.122662 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:53:54.123816 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:54.123457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40999 (local address 127.0.0.1:47596)
0929 02:53:54.123606 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:54.123680 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 02:53:54.204756 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:54.204261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40999 (local address 127.0.0.1:47610)
0929 02:53:54.204465 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:54.204565 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 02:53:54.204843 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 22914
I20260929 02:53:54.206812 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:54.206367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:40999 (local address 127.0.0.1:47616)
0929 02:53:54.206581 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:54.206674 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:40999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 02:53:54.217731 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 22563
I20260929 02:53:54.218717 22624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:54.326218 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 22563
2026-09-29T02:53:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2670 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:53:54 dist-test-slave-t6c1 krb5kdc[23094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:53:54 dist-test-slave-t6c1 krb5kdc[23094](info): set up 2 sockets
Sep 29 02:53:54 dist-test-slave-t6c1 krb5kdc[23094](info): commencing operation
krb5kdc: starting...
W20260929 02:53:55.335117 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:53:55 dist-test-slave-t6c1 krb5kdc[23094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:53:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:53:55.574081 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45981
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:53:55.902412 23107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:55.902694 23107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:55.902817 23107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:55.910033 23107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:53:55.910146 23107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:55.910204 23107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:55.910290 23107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:53:55.910351 23107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:53:55.920352 23107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:55.921659 23107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:55.922984 23107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:55.933947 23113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:55.933991 23115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:55.934021 23112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:55.934756 23107 server_base.cc:1034] running on GCE node
I20260929 02:53:55.935531 23107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:55.936888 23107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:55.938095 23107 hybrid_clock.cc:648] HybridClock initialized: now 1790650435938050 us; error 32 us; skew 500 ppm
I20260929 02:53:55.938422 23107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:55 dist-test-slave-t6c1 krb5kdc[23094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650435, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:55.941419 23107 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:53:55.942978 23107 webserver.cc:460] Webserver started at http://127.4.197.190:36005/ using document root <none> and password file <none>
I20260929 02:53:55.943655 23107 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:55.943753 23107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:55.944006 23107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:55.947114 23107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "bc981b75955f41e1bda2cb88705076dd"
format_stamp: "Formatted at 2026-09-29 02:53:55 on dist-test-slave-t6c1"
server_key: "a9582ece3234c74e98432d9270efd804"
server_key_iv: "18a979b22681c565a48337d7b9e73fd2"
server_key_version: "encryptionkey@0"
I20260929 02:53:55.947841 23107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "bc981b75955f41e1bda2cb88705076dd"
format_stamp: "Formatted at 2026-09-29 02:53:55 on dist-test-slave-t6c1"
server_key: "a9582ece3234c74e98432d9270efd804"
server_key_iv: "18a979b22681c565a48337d7b9e73fd2"
server_key_version: "encryptionkey@0"
I20260929 02:53:55.953001 23107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 02:53:55.956513 23122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:55.957967 23107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:53:55.958117 23107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "bc981b75955f41e1bda2cb88705076dd"
format_stamp: "Formatted at 2026-09-29 02:53:55 on dist-test-slave-t6c1"
server_key: "a9582ece3234c74e98432d9270efd804"
server_key_iv: "18a979b22681c565a48337d7b9e73fd2"
server_key_version: "encryptionkey@0"
I20260929 02:53:55.958300 23107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:55.974831 23107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:55.979075 23107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:55.979307 23107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:55.996407 23107 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:45981
I20260929 02:53:55.996410 23173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:45981 every 8 connection(s)
I20260929 02:53:55.997836 23107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:53:56.001745 23174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:56.007459 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 23107
I20260929 02:53:56.007634 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:53:56.008184 4886 external_mini_cluster.cc:1428] Setting key 837204e4181eed64b26907b85ac5f22e
I20260929 02:53:56.012568 23174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd: Bootstrap starting.
I20260929 02:53:56.016074 23174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:56.017274 23174 log.cc:824] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:56.020560 23174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd: No bootstrap required, opened a new log
Sep 29 02:53:56 dist-test-slave-t6c1 krb5kdc[23094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:56.028424 23174 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc981b75955f41e1bda2cb88705076dd" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45981 } }
I20260929 02:53:56.028790 23174 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:56.028892 23174 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc981b75955f41e1bda2cb88705076dd, State: Initialized, Role: FOLLOWER
I20260929 02:53:56.029592 23174 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc981b75955f41e1bda2cb88705076dd" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45981 } }
I20260929 02:53:56.029881 23174 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:53:56.030017 23174 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:53:56.030185 23174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:56.031183 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:56.010023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.4.197.190:45981)
0929 02:53:56.010782 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:56.010793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:56.010834 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:53:56.012512 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:56.012534 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:56.012935 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:56.013255 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:56.015073 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:56.016372 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:56.018145 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:56.018967 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:56.019097 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:56.023788 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:56.023795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:56.023812 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:56.023854 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:56.026847 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:56.027525 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:56.027531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:56.027537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:56.027713 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:56.028084 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:56.028088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:56.028090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:56.028609 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:56.028876 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:56.029309 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 02:53:56.029746 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":346,"threads_started":1}
I20260929 02:53:56.032840 23174 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc981b75955f41e1bda2cb88705076dd" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45981 } }
I20260929 02:53:56.033308 23174 leader_election.cc:302] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc981b75955f41e1bda2cb88705076dd; no voters:
I20260929 02:53:56.033928 23174 leader_election.cc:288] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:53:56.034051 23179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:53:56.034636 23179 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [term 1 LEADER]: Becoming Leader. State: Replica: bc981b75955f41e1bda2cb88705076dd, State: Running, Role: LEADER
I20260929 02:53:56.035238 23179 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc981b75955f41e1bda2cb88705076dd" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45981 } }
I20260929 02:53:56.036101 23174 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:53:56.038158 23180 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc981b75955f41e1bda2cb88705076dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc981b75955f41e1bda2cb88705076dd" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45981 } } }
I20260929 02:53:56.038172 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc981b75955f41e1bda2cb88705076dd. Latest consensus state: current_term: 1 leader_uuid: "bc981b75955f41e1bda2cb88705076dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc981b75955f41e1bda2cb88705076dd" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45981 } } }
I20260929 02:53:56.038606 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:56.038614 23180 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:56.040195 23185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:53:56.045524 23185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:53:56.054560 23185 catalog_manager.cc:1347] Generated new cluster ID: 786c4e2ceb7b4700b8205c57790edbfb
I20260929 02:53:56.054687 23185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:53:56.071859 23185 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:53:56.073676 23185 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:53:56.082789 23185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd: Generated new TSK 0
I20260929 02:53:56.083984 23185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:53:56.150844 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45981
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:53:56.481264 23200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:56.481581 23200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:56.481719 23200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:56.489291 23200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:56.489432 23200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:56.489602 23200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:53:56.499646 23200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:56.500970 23200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:56.502281 23200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:56.514577 23205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:56.514539 23208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:56.514608 23206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:56.515570 23200 server_base.cc:1034] running on GCE node
I20260929 02:53:56.516096 23200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:56.516937 23200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:56.518121 23200 hybrid_clock.cc:648] HybridClock initialized: now 1790650436518080 us; error 34 us; skew 500 ppm
I20260929 02:53:56.518399 23200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:56 dist-test-slave-t6c1 krb5kdc[23094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650436, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:56.521230 23200 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:53:56.522488 23200 webserver.cc:460] Webserver started at http://127.4.197.129:43897/ using document root <none> and password file <none>
I20260929 02:53:56.523183 23200 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:56.523293 23200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:56.523579 23200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:56.526429 23200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "85b49dc692d148b0900f64ffe63a3161"
format_stamp: "Formatted at 2026-09-29 02:53:56 on dist-test-slave-t6c1"
server_key: "4494b7af8ad35ffbe280298fee1502c8"
server_key_iv: "d38c5aec4ade5d1b35d64dfc5db8b577"
server_key_version: "encryptionkey@0"
I20260929 02:53:56.527220 23200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "85b49dc692d148b0900f64ffe63a3161"
format_stamp: "Formatted at 2026-09-29 02:53:56 on dist-test-slave-t6c1"
server_key: "4494b7af8ad35ffbe280298fee1502c8"
server_key_iv: "d38c5aec4ade5d1b35d64dfc5db8b577"
server_key_version: "encryptionkey@0"
I20260929 02:53:56.532482 23200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:53:56.535775 23215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:56.537189 23200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 02:53:56.537326 23200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "85b49dc692d148b0900f64ffe63a3161"
format_stamp: "Formatted at 2026-09-29 02:53:56 on dist-test-slave-t6c1"
server_key: "4494b7af8ad35ffbe280298fee1502c8"
server_key_iv: "d38c5aec4ade5d1b35d64dfc5db8b577"
server_key_version: "encryptionkey@0"
I20260929 02:53:56.537513 23200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:56.557147 23200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:56.561383 23200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:56.561679 23200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:56.562665 23200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:56.564071 23200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:56.564142 23200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:56.564235 23200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:56.564277 23200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:56.594040 23200 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:39495
I20260929 02:53:56.594076 23327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:39495 every 8 connection(s)
I20260929 02:53:56.595361 23200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:53:56.597244 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 23200
I20260929 02:53:56.597424 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:53:56.597865 4886 external_mini_cluster.cc:1428] Setting key 6ebe9d85a0f975d1c8aa03a5c43f28e2
Sep 29 02:53:56 dist-test-slave-t6c1 krb5kdc[23094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650436, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:56.615355 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:56.598051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:54767 (local address 127.4.197.190:45981)
0929 02:53:56.598510 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:56.598516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:56.599933 (+ 1417us) server_negotiation.cc:407] Connection header received
0929 02:53:56.601527 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:56.601532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:56.601616 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:56.601713 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:56.603276 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:56.603976 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:56.605498 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:56.605907 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:56.606016 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:56.609772 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:56.609778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:56.609782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:56.609818 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:56.612587 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:56.613246 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:56.613251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:56.613254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:56.613391 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:56.613744 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:56.613748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:56.613750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:56.614044 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:56.614172 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:56.614774 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 02:53:56.615008 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":225,"threads_started":1}
I20260929 02:53:56.616108 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:56.598668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45981 (local address 127.4.197.129:54767)
0929 02:53:56.599475 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:56.599530 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:56.601261 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:56.601972 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:56.601980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:56.602497 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:56.603119 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:56.603139 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:56.604130 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:56.604133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:56.605366 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:56.605379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:56.606116 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:56.606123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:56.606348 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:56.607019 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:53:56.607045 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:56.609559 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:56.612715 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:56.612721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:56.612733 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:56.613135 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:56.613477 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:56.613480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:56.613482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:56.613650 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:56.614141 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:56.614146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:56.614442 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 02:53:56.614649 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 02:53:56.615061 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":298,"threads_started":1}
I20260929 02:53:56.617512 23328 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45981
I20260929 02:53:56.617830 23328 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:56.618688 23328 heartbeater.cc:511] Master 127.4.197.190:45981 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:53:56.620787 23139 ts_manager.cc:194] Registered new tserver with Master: 85b49dc692d148b0900f64ffe63a3161 (127.4.197.129:39495)
I20260929 02:53:56.622277 23139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:54767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:53:56.650336 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45981
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:53:56.968598 23334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:56.968899 23334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:56.968995 23334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:56.976459 23334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:56.976596 23334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:56.976724 23334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:53:56.987157 23334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:56.988457 23334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:56.989779 23334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:57.002429 23340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:57.002451 23339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:57.003257 23342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:57.003587 23334 server_base.cc:1034] running on GCE node
I20260929 02:53:57.004163 23334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:57.005059 23334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:57.006282 23334 hybrid_clock.cc:648] HybridClock initialized: now 1790650437006212 us; error 41 us; skew 500 ppm
I20260929 02:53:57.006569 23334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:57 dist-test-slave-t6c1 krb5kdc[23094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:57.009608 23334 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:53:57.011062 23334 webserver.cc:460] Webserver started at http://127.4.197.130:36671/ using document root <none> and password file <none>
I20260929 02:53:57.011725 23334 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:57.011842 23334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:57.012121 23334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:57.014809 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "7fdeca03c0e640c8b9ba985f3a770027"
format_stamp: "Formatted at 2026-09-29 02:53:57 on dist-test-slave-t6c1"
server_key: "983e0be6fc8fec185f18b0c0125c5df1"
server_key_iv: "febe0f112a2436ad13e26710b052efad"
server_key_version: "encryptionkey@0"
I20260929 02:53:57.015578 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "7fdeca03c0e640c8b9ba985f3a770027"
format_stamp: "Formatted at 2026-09-29 02:53:57 on dist-test-slave-t6c1"
server_key: "983e0be6fc8fec185f18b0c0125c5df1"
server_key_iv: "febe0f112a2436ad13e26710b052efad"
server_key_version: "encryptionkey@0"
I20260929 02:53:57.020309 23334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:53:57.023677 23349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:57.025077 23334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:53:57.025225 23334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "7fdeca03c0e640c8b9ba985f3a770027"
format_stamp: "Formatted at 2026-09-29 02:53:57 on dist-test-slave-t6c1"
server_key: "983e0be6fc8fec185f18b0c0125c5df1"
server_key_iv: "febe0f112a2436ad13e26710b052efad"
server_key_version: "encryptionkey@0"
I20260929 02:53:57.025405 23334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:57.041301 23334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:57.045527 23334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:57.045799 23334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:57.046774 23334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:57.048172 23334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:57.048250 23334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:57.048377 23334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:57.048429 23334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:57.077827 23334 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:39205
I20260929 02:53:57.077879 23461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:39205 every 8 connection(s)
I20260929 02:53:57.079196 23334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:53:57.082901 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 23334
I20260929 02:53:57.083124 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:53:57.083649 4886 external_mini_cluster.cc:1428] Setting key b21421ccd6a5c63275329aea387677db
Sep 29 02:53:57 dist-test-slave-t6c1 krb5kdc[23094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:57.101588 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.081642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:38625 (local address 127.4.197.190:45981)
0929 02:53:57.081875 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:57.081888 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:57.083132 (+ 1244us) server_negotiation.cc:407] Connection header received
0929 02:53:57.085242 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:57.085246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:57.085316 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:57.085423 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:57.087233 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.087949 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.089533 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.089963 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.090076 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.096205 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:57.096212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:57.096216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:57.096254 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:57.098679 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:57.099406 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:57.099410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:57.099412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:57.099537 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:57.099940 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:57.099944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:57.099946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:57.100198 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:57.100330 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:57.101015 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 02:53:57.101242 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:53:57.102556 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.082174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45981 (local address 127.4.197.130:38625)
0929 02:53:57.082913 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:57.083010 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:57.084953 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:57.085715 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:57.085727 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:57.086297 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:57.087048 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.087069 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.088107 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.088112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.089385 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.089398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.090187 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.090195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.090436 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.093665 (+ 3229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:53:57.093690 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:57.095974 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:57.098809 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:57.098819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:57.098836 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:57.099279 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:57.099638 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:57.099643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:57.099647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:57.099826 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:57.100316 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:57.100323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:57.100664 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 02:53:57.100901 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 02:53:57.101420 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":299,"threads_started":1}
I20260929 02:53:57.104004 23462 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45981
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
I20260929 02:53:57.104319 23462 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:57.105052 23462 heartbeater.cc:511] Master 127.4.197.190:45981 requested a full tablet report, sending...
I20260929 02:53:57.106369 23139 ts_manager.cc:194] Registered new tserver with Master: 7fdeca03c0e640c8b9ba985f3a770027 (127.4.197.130:39205)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
I20260929 02:53:57.107198 23139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:38625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:53:57.130326 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45981
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:53:57.448320 23467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:57.448591 23467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:57.448709 23467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:57.456001 23467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:57.456135 23467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:57.456290 23467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:53:57.466516 23467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:45981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:57.467854 23467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:57.469157 23467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:57.481673 23472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:57.481717 23473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:57.481750 23475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:57.482667 23467 server_base.cc:1034] running on GCE node
I20260929 02:53:57.483341 23467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:57.484345 23467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:57.485569 23467 hybrid_clock.cc:648] HybridClock initialized: now 1790650437485509 us; error 52 us; skew 500 ppm
I20260929 02:53:57.485888 23467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:53:57 dist-test-slave-t6c1 krb5kdc[23094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:53:57.488914 23467 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:53:57.490235 23467 webserver.cc:460] Webserver started at http://127.4.197.131:36981/ using document root <none> and password file <none>
I20260929 02:53:57.490871 23467 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:57.491010 23467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:57.491298 23467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:57.494035 23467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "a0fdec95544242ccbd05d9a637eb599b"
format_stamp: "Formatted at 2026-09-29 02:53:57 on dist-test-slave-t6c1"
server_key: "77b0527de324ff5ec401844f91d69264"
server_key_iv: "4e17f1ee180d72b48301eacbbdcd672e"
server_key_version: "encryptionkey@0"
I20260929 02:53:57.494771 23467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "a0fdec95544242ccbd05d9a637eb599b"
format_stamp: "Formatted at 2026-09-29 02:53:57 on dist-test-slave-t6c1"
server_key: "77b0527de324ff5ec401844f91d69264"
server_key_iv: "4e17f1ee180d72b48301eacbbdcd672e"
server_key_version: "encryptionkey@0"
I20260929 02:53:57.499559 23467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 02:53:57.502974 23482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:57.504446 23467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:53:57.504592 23467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "a0fdec95544242ccbd05d9a637eb599b"
format_stamp: "Formatted at 2026-09-29 02:53:57 on dist-test-slave-t6c1"
server_key: "77b0527de324ff5ec401844f91d69264"
server_key_iv: "4e17f1ee180d72b48301eacbbdcd672e"
server_key_version: "encryptionkey@0"
I20260929 02:53:57.504777 23467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:57.518800 23467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:57.522462 23467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:57.522701 23467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:57.523676 23467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:57.524906 23467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:57.524976 23467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:57.525054 23467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:57.525098 23467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:57.553844 23467 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:36135
I20260929 02:53:57.553881 23594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:36135 every 8 connection(s)
I20260929 02:53:57.555119 23467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:53:57.563148 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 23467
I20260929 02:53:57.563305 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:53:57.563748 4886 external_mini_cluster.cc:1428] Setting key 5d9a7857c90ed574ee2bae65bbfcb84e
Sep 29 02:53:57 dist-test-slave-t6c1 krb5kdc[23094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:53:57.575055 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.557552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:57879 (local address 127.4.197.190:45981)
0929 02:53:57.557780 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:57.557787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:57.558958 (+ 1171us) server_negotiation.cc:407] Connection header received
0929 02:53:57.560824 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:57.560829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:57.560927 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:57.561023 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:57.562806 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.563704 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.565818 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.566459 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.566582 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.569945 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:57.569950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:57.569953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:57.569987 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:57.572475 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:57.573106 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:57.573109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:57.573112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:57.573233 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:57.573577 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:57.573580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:57.573582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:57.573839 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:57.573955 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:57.574570 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 02:53:57.574738 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:53:57.576164 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.558061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45981 (local address 127.4.197.131:57879)
0929 02:53:57.558768 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:57.558806 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:57.560530 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:57.561272 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:57.561281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:57.561938 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:57.562600 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.562621 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.563877 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.563882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.565693 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.565706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.566683 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.566692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.566907 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.567588 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:53:57.567612 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:57.569741 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:57.572592 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:57.572599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:57.572612 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:57.573020 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:57.573308 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:53:57.573311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:53:57.573313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:53:57.573502 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:53:57.573933 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:57.573938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:57.574255 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 02:53:57.574502 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 02:53:57.574889 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":273,"threads_started":1}
I20260929 02:53:57.577699 23595 heartbeater.cc:347] Connected to a master server at 127.4.197.190:45981
I20260929 02:53:57.578033 23595 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:57.578791 23595 heartbeater.cc:511] Master 127.4.197.190:45981 requested a full tablet report, sending...
I20260929 02:53:57.580163 23139 ts_manager.cc:194] Registered new tserver with Master: a0fdec95544242ccbd05d9a637eb599b (127.4.197.131:36135)
I20260929 02:53:57.580986 23139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:57879
I20260929 02:53:57.590870 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:53:57.608518 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.595007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.4.197.190:45981)
0929 02:53:57.595210 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:57.595217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:57.595631 (+ 414us) server_negotiation.cc:407] Connection header received
0929 02:53:57.595795 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:57.595799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:57.595872 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:57.595955 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:57.597829 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.598667 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.600315 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.600707 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.600799 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.603292 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:57.603316 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:57.603319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:57.603354 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:57.605808 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:57.606526 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:57.606530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:57.606533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:57.606679 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:57.607077 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:57.607081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:57.607083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:57.607400 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:57.607625 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:57.607974 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 02:53:57.608188 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:53:57.612530 23139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:53:57.619140 23139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:53:57.625554 23328 heartbeater.cc:503] Master 127.4.197.190:45981 was elected leader, sending a full tablet report...
I20260929 02:53:57.655896 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.631055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:39205 (local address 127.0.0.1:54922)
0929 02:53:57.631951 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:57.632003 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:57.632176 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:57.647395 (+ 15219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:57.647403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:57.647441 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:57.647984 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.648002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.650141 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.650147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.652594 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.652609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.654674 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.654680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.654901 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.654998 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 02:53:57.655121 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 02:53:57.655418 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":190,"thread_start_us":16298,"threads_started":1}
I20260929 02:53:57.656086 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.631250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.4.197.130:39205)
0929 02:53:57.646478 (+ 15228us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:57.646487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:57.646512 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:53:57.646617 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:57.646626 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:57.646908 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:57.648224 (+ 1316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:57.648458 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.649980 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.652745 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.654120 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.655337 (+ 1217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.655538 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:57.655618 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 02:53:57.655785 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15046,"thread_start_us":15831,"threads_started":1}
I20260929 02:53:57.655995 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.631198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:39495 (local address 127.0.0.1:34616)
0929 02:53:57.646186 (+ 14988us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:57.646211 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:57.646368 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:57.647627 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:57.647630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:57.647651 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:57.648143 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.648155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.650141 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.650147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.653198 (+ 3051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.653214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.655093 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.655100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.655275 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.655304 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:53:57.655390 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 02:53:57.655519 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14819,"mutex_wait_us":68,"thread_start_us":640,"threads_started":1}
I20260929 02:53:57.656618 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.631672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.4.197.129:39495)
0929 02:53:57.644101 (+ 12429us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:57.644111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:57.646516 (+ 2405us) server_negotiation.cc:407] Connection header received
0929 02:53:57.646617 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:57.646626 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:57.646913 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:57.647141 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:57.648332 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.649984 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.653363 (+ 3379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.654811 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.654962 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.655196 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:57.656135 (+ 939us) server_negotiation.cc:299] Negotiation successful
0929 02:53:57.656306 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12212,"thread_start_us":13596,"threads_started":1}
I20260929 02:53:57.659960 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.633440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53070 (local address 127.4.197.131:36135)
0929 02:53:57.651664 (+ 18224us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:57.651673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:57.651710 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:53:57.651907 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:57.651916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:57.652207 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:57.652429 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:57.653078 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.654590 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.658041 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.658985 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.659098 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.659317 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:57.659506 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 02:53:57.659655 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18052,"thread_start_us":17674,"threads_started":1}
I20260929 02:53:57.659958 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.633157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:36135 (local address 127.0.0.1:53070)
0929 02:53:57.651579 (+ 18422us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:57.651603 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:57.651727 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:57.652430 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:57.652435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:57.652462 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:57.652923 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.652931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.654763 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.654772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.657895 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.657907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.659171 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.659177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.659335 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.659367 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:53:57.659446 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:53:57.659575 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18276,"mutex_wait_us":36,"thread_start_us":18118,"threads_started":1}
I20260929 02:53:57.660497 23263 tablet_service.cc:1467] Processing CreateTablet for tablet f4ed76497b2340288e22e6caf2a8e2e6 (DEFAULT_TABLE table=test-table [id=a86c3fe76f014e40a4bd5a58f9fbde7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:57.660497 23397 tablet_service.cc:1467] Processing CreateTablet for tablet f4ed76497b2340288e22e6caf2a8e2e6 (DEFAULT_TABLE table=test-table [id=a86c3fe76f014e40a4bd5a58f9fbde7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:57.662016 23397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ed76497b2340288e22e6caf2a8e2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:57.662016 23263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ed76497b2340288e22e6caf2a8e2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:57.670065 23530 tablet_service.cc:1467] Processing CreateTablet for tablet f4ed76497b2340288e22e6caf2a8e2e6 (DEFAULT_TABLE table=test-table [id=a86c3fe76f014e40a4bd5a58f9fbde7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:53:57.671728 23530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ed76497b2340288e22e6caf2a8e2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:57.677436 23614 tablet_bootstrap.cc:492] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: Bootstrap starting.
I20260929 02:53:57.678215 23613 tablet_bootstrap.cc:492] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: Bootstrap starting.
I20260929 02:53:57.682075 23614 tablet_bootstrap.cc:654] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:57.682145 23613 tablet_bootstrap.cc:654] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:57.683223 23614 log.cc:824] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:57.683403 23613 log.cc:824] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:57.686053 23614 tablet_bootstrap.cc:492] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: No bootstrap required, opened a new log
I20260929 02:53:57.686340 23614 ts_tablet_manager.cc:1397] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:53:57.686565 23613 tablet_bootstrap.cc:492] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: No bootstrap required, opened a new log
I20260929 02:53:57.686832 23613 ts_tablet_manager.cc:1397] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:53:57.687171 23617 tablet_bootstrap.cc:492] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: Bootstrap starting.
I20260929 02:53:57.690811 23617 tablet_bootstrap.cc:654] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:57.692118 23617 log.cc:824] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:57.694693 23617 tablet_bootstrap.cc:492] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: No bootstrap required, opened a new log
I20260929 02:53:57.694957 23617 ts_tablet_manager.cc:1397] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:53:57.695438 23613 raft_consensus.cc:348] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 } } peers { permanent_uuid: "85b49dc692d148b0900f64ffe63a3161" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 39495 } } peers { permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 } }
I20260929 02:53:57.695869 23613 raft_consensus.cc:374] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:57.695959 23613 raft_consensus.cc:729] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b49dc692d148b0900f64ffe63a3161, State: Initialized, Role: FOLLOWER
I20260929 02:53:57.696571 23613 consensus_queue.cc:260] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 } } peers { permanent_uuid: "85b49dc692d148b0900f64ffe63a3161" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 39495 } } peers { permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 } }
I20260929 02:53:57.697305 23614 raft_consensus.cc:348] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 } } peers { permanent_uuid: "85b49dc692d148b0900f64ffe63a3161" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 39495 } } peers { permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 } }
I20260929 02:53:57.697703 23614 raft_consensus.cc:374] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:57.697793 23614 raft_consensus.cc:729] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fdeca03c0e640c8b9ba985f3a770027, State: Initialized, Role: FOLLOWER
I20260929 02:53:57.698264 23613 ts_tablet_manager.cc:1428] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 02:53:57.698448 23614 consensus_queue.cc:260] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 } } peers { permanent_uuid: "85b49dc692d148b0900f64ffe63a3161" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 39495 } } peers { permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 } }
I20260929 02:53:57.699468 23462 heartbeater.cc:503] Master 127.4.197.190:45981 was elected leader, sending a full tablet report...
I20260929 02:53:57.700022 23614 ts_tablet_manager.cc:1428] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20260929 02:53:57.703863 23617 raft_consensus.cc:348] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 } } peers { permanent_uuid: "85b49dc692d148b0900f64ffe63a3161" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 39495 } } peers { permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 } }
I20260929 02:53:57.704246 23617 raft_consensus.cc:374] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:57.704345 23617 raft_consensus.cc:729] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0fdec95544242ccbd05d9a637eb599b, State: Initialized, Role: FOLLOWER
I20260929 02:53:57.704878 23617 consensus_queue.cc:260] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 } } peers { permanent_uuid: "85b49dc692d148b0900f64ffe63a3161" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 39495 } } peers { permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 } }
I20260929 02:53:57.705703 23595 heartbeater.cc:503] Master 127.4.197.190:45981 was elected leader, sending a full tablet report...
I20260929 02:53:57.706179 23617 ts_tablet_manager.cc:1428] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260929 02:53:57.806568 23596 tablet.cc:2369] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:53:57.831080 23463 tablet.cc:2369] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:53:57.839179 23619 raft_consensus.cc:482] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:53:57.839460 23619 raft_consensus.cc:504] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 } } peers { permanent_uuid: "85b49dc692d148b0900f64ffe63a3161" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 39495 } } peers { permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 } }
I20260929 02:53:57.841303 23619 leader_election.cc:288] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fdeca03c0e640c8b9ba985f3a770027 (127.4.197.130:39205), a0fdec95544242ccbd05d9a637eb599b (127.4.197.131:36135)
W20260929 02:53:57.847601 23329 tablet.cc:2369] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:53:57.847955 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.841952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:58453 (local address 127.4.197.131:36135)
0929 02:53:57.842150 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:57.842157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:57.842780 (+ 623us) server_negotiation.cc:407] Connection header received
0929 02:53:57.843007 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:57.843012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:57.843075 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:57.843148 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:57.843841 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.844591 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.846259 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.847065 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.847207 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.847285 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:57.847568 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 02:53:57.847695 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:53:57.848097 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.841742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:36135 (local address 127.4.197.129:58453)
0929 02:53:57.842670 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:57.842684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:57.842861 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:57.843170 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:57.843174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:57.843216 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:57.843693 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.843703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.844726 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.844730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.846118 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.846128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.847263 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.847269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.847417 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.847445 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:53:57.847536 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:53:57.847676 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":19,"spinlock_wait_cycles":58624,"thread_start_us":570,"threads_started":1}
I20260929 02:53:57.848516 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.841507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:39205 (local address 127.4.197.129:37475)
0929 02:53:57.842629 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:57.842646 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:57.842800 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:57.843151 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:57.843155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:57.843190 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:53:57.843627 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.843641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.844599 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.844604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.846103 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:57.846115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.847448 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.847455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:57.847636 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.847669 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:53:57.847821 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 02:53:57.847950 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":45,"thread_start_us":711,"threads_started":1}
I20260929 02:53:57.848953 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.841739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:37475 (local address 127.4.197.130:39205)
0929 02:53:57.841975 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:57.841982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:57.842747 (+ 765us) server_negotiation.cc:407] Connection header received
0929 02:53:57.842975 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:57.842982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:57.843047 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:57.843124 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:53:57.843774 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.844478 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.846251 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.847066 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.847163 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.847218 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:57.848565 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0929 02:53:57.848710 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:53:57.849133 23550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ed76497b2340288e22e6caf2a8e2e6" candidate_uuid: "85b49dc692d148b0900f64ffe63a3161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0fdec95544242ccbd05d9a637eb599b" is_pre_election: true
I20260929 02:53:57.849705 23550 raft_consensus.cc:2454] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b49dc692d148b0900f64ffe63a3161 in term 0.
I20260929 02:53:57.849821 23417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ed76497b2340288e22e6caf2a8e2e6" candidate_uuid: "85b49dc692d148b0900f64ffe63a3161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fdeca03c0e640c8b9ba985f3a770027" is_pre_election: true
I20260929 02:53:57.850317 23417 raft_consensus.cc:2454] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b49dc692d148b0900f64ffe63a3161 in term 0.
I20260929 02:53:57.850349 23216 leader_election.cc:302] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85b49dc692d148b0900f64ffe63a3161, a0fdec95544242ccbd05d9a637eb599b; no voters:
I20260929 02:53:57.850754 23619 raft_consensus.cc:2781] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:53:57.850912 23619 raft_consensus.cc:482] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:53:57.851065 23619 raft_consensus.cc:3037] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:57.853235 23619 raft_consensus.cc:504] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 } } peers { permanent_uuid: "85b49dc692d148b0900f64ffe63a3161" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 39495 } } peers { permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 } }
I20260929 02:53:57.853858 23619 leader_election.cc:288] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [CANDIDATE]: Term 1 election: Requested vote from peers 7fdeca03c0e640c8b9ba985f3a770027 (127.4.197.130:39205), a0fdec95544242ccbd05d9a637eb599b (127.4.197.131:36135)
I20260929 02:53:57.854377 23417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ed76497b2340288e22e6caf2a8e2e6" candidate_uuid: "85b49dc692d148b0900f64ffe63a3161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fdeca03c0e640c8b9ba985f3a770027"
I20260929 02:53:57.854462 23550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ed76497b2340288e22e6caf2a8e2e6" candidate_uuid: "85b49dc692d148b0900f64ffe63a3161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0fdec95544242ccbd05d9a637eb599b"
I20260929 02:53:57.854569 23417 raft_consensus.cc:3037] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:57.854668 23550 raft_consensus.cc:3037] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:57.856841 23417 raft_consensus.cc:2454] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b49dc692d148b0900f64ffe63a3161 in term 1.
I20260929 02:53:57.856842 23550 raft_consensus.cc:2454] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b49dc692d148b0900f64ffe63a3161 in term 1.
I20260929 02:53:57.857277 23217 leader_election.cc:302] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fdeca03c0e640c8b9ba985f3a770027, 85b49dc692d148b0900f64ffe63a3161; no voters:
I20260929 02:53:57.857583 23619 raft_consensus.cc:2781] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:53:57.858006 23619 raft_consensus.cc:686] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 1 LEADER]: Becoming Leader. State: Replica: 85b49dc692d148b0900f64ffe63a3161, State: Running, Role: LEADER
I20260929 02:53:57.858474 23619 consensus_queue.cc:237] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 } } peers { permanent_uuid: "85b49dc692d148b0900f64ffe63a3161" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 39495 } } peers { permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 } }
I20260929 02:53:57.863415 23138 catalog_manager.cc:5495] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85b49dc692d148b0900f64ffe63a3161 (127.4.197.129). New cstate: current_term: 1 leader_uuid: "85b49dc692d148b0900f64ffe63a3161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b49dc692d148b0900f64ffe63a3161" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 39495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 } health_report { overall_health: UNKNOWN } } }
I20260929 02:53:57.889816 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.884775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.4.197.129:39495)
0929 02:53:57.884988 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:57.884994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:57.885060 (+ 66us) server_negotiation.cc:407] Connection header received
0929 02:53:57.885225 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:57.885229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:57.885318 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:57.885445 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:53:57.885750 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.886447 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.887758 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.888194 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.888321 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.888762 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:53:57.889299 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:53:57.889431 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:57.889503 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 02:53:57.889545 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:53:57.897753 23550 raft_consensus.cc:1261] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b [term 1 FOLLOWER]: Refusing update from remote peer 85b49dc692d148b0900f64ffe63a3161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:53:57.897753 23417 raft_consensus.cc:1261] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027 [term 1 FOLLOWER]: Refusing update from remote peer 85b49dc692d148b0900f64ffe63a3161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:53:57.898630 23619 consensus_queue.cc:1035] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fdeca03c0e640c8b9ba985f3a770027" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:53:57.898905 23624 consensus_queue.cc:1035] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0fdec95544242ccbd05d9a637eb599b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:53:57.910104 23627 mvcc.cc:205] Tried to move back new op lower bound from 7334504193619398656 to 7334504193472139264. Current Snapshot: MvccSnapshot[applied={T|T < 7334504193619398656}]
I20260929 02:53:57.922640 23139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:53:57.922907 23139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:53:57.926542 23139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc981b75955f41e1bda2cb88705076dd: Sending DeleteTablet for 3 replicas of tablet f4ed76497b2340288e22e6caf2a8e2e6
I20260929 02:53:57.927758 23263 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ed76497b2340288e22e6caf2a8e2e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:57 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:34616
I20260929 02:53:57.927750 23530 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ed76497b2340288e22e6caf2a8e2e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:57 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:53070
I20260929 02:53:57.927786 23397 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ed76497b2340288e22e6caf2a8e2e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:53:57 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:54922
I20260929 02:53:57.932477 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 23200
I20260929 02:53:57.940560 23637 tablet_replica.cc:330] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: stopping tablet replica
I20260929 02:53:57.941074 23637 raft_consensus.cc:2229] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:53:57.941612 23637 raft_consensus.cc:2258] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:57.942440 23638 tablet_replica.cc:330] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: stopping tablet replica
I20260929 02:53:57.942891 23638 raft_consensus.cc:2229] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:53:57.943610 23638 raft_consensus.cc:2258] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:57.946199 23638 ts_tablet_manager.cc:1905] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:57.949357 23639 tablet_replica.cc:330] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: stopping tablet replica
I20260929 02:53:57.949760 23639 raft_consensus.cc:2229] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:53:57.950276 23639 raft_consensus.cc:2258] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:53:57.950835 23637 ts_tablet_manager.cc:1905] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:57.952394 23639 ts_tablet_manager.cc:1905] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:53:57.954558 23638 ts_tablet_manager.cc:1918] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:57.954732 23638 log.cc:1196] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/f4ed76497b2340288e22e6caf2a8e2e6
I20260929 02:53:57.955281 23638 ts_tablet_manager.cc:1939] T f4ed76497b2340288e22e6caf2a8e2e6 P 85b49dc692d148b0900f64ffe63a3161: Deleting consensus metadata
I20260929 02:53:57.957160 23125 catalog_manager.cc:4841] TS 85b49dc692d148b0900f64ffe63a3161 (127.4.197.129:39495): tablet f4ed76497b2340288e22e6caf2a8e2e6 (table test-table [id=a86c3fe76f014e40a4bd5a58f9fbde7b]) successfully deleted
I20260929 02:53:57.958604 23637 ts_tablet_manager.cc:1918] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:57.958842 23637 log.cc:1196] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/f4ed76497b2340288e22e6caf2a8e2e6
I20260929 02:53:57.959003 23639 ts_tablet_manager.cc:1918] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:53:57.959156 23639 log.cc:1196] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/f4ed76497b2340288e22e6caf2a8e2e6
I20260929 02:53:57.959575 23637 ts_tablet_manager.cc:1939] T f4ed76497b2340288e22e6caf2a8e2e6 P a0fdec95544242ccbd05d9a637eb599b: Deleting consensus metadata
I20260929 02:53:57.959625 23639 ts_tablet_manager.cc:1939] T f4ed76497b2340288e22e6caf2a8e2e6 P 7fdeca03c0e640c8b9ba985f3a770027: Deleting consensus metadata
I20260929 02:53:57.961447 23124 catalog_manager.cc:4841] TS 7fdeca03c0e640c8b9ba985f3a770027 (127.4.197.130:39205): tablet f4ed76497b2340288e22e6caf2a8e2e6 (table test-table [id=a86c3fe76f014e40a4bd5a58f9fbde7b]) successfully deleted
I20260929 02:53:57.961584 23123 catalog_manager.cc:4841] TS a0fdec95544242ccbd05d9a637eb599b (127.4.197.131:36135): tablet f4ed76497b2340288e22e6caf2a8e2e6 (table test-table [id=a86c3fe76f014e40a4bd5a58f9fbde7b]) successfully deleted
Sep 29 02:53:57 dist-test-slave-t6c1 krb5kdc[23094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:53:57.973085 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:57.934359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.4.197.129:39495)
0929 02:53:57.934547 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:57.934559 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:57.948452 (+ 13893us) server_negotiation.cc:407] Connection header received
0929 02:53:57.948652 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:57.948657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:57.948736 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:57.948834 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:57.958638 (+ 9804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.959864 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.961807 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:57.962300 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:57.962403 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:57.967411 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:57.967424 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:57.967440 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:57.967476 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:57.970389 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:57.971202 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:57.971209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:57.971211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:57.971358 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:57.971736 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:57.971739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:57.971741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:57.972070 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:57.972305 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:57.972624 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 02:53:57.972797 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 02:53:57.973766 23323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:58.075824 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 23200
I20260929 02:53:58.091001 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 23334
Sep 29 02:53:58 dist-test-slave-t6c1 krb5kdc[23094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:53:58.106529 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:58.091826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.4.197.130:39205)
0929 02:53:58.092019 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:58.092023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:58.092227 (+ 204us) server_negotiation.cc:407] Connection header received
0929 02:53:58.092431 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:58.092435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:58.092506 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:58.092601 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:58.094368 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:58.095283 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:58.096799 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:58.097169 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:58.097256 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:58.101457 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:58.101470 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:58.101486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:58.101521 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:58.104278 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:58.104891 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:58.104897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:58.104899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:58.105027 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:58.105304 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:58.105307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:58.105309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:58.105652 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:58.105883 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:58.106072 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 02:53:58.106228 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:53:58.107148 23457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:58.194361 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 23334
I20260929 02:53:58.208190 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 23467
Sep 29 02:53:58 dist-test-slave-t6c1 krb5kdc[23094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:53:58.223220 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:58.209003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.4.197.131:36135)
0929 02:53:58.209187 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:58.209192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:58.209211 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:53:58.209371 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:58.209375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:58.209460 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:58.209545 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:58.210802 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:58.211571 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:58.212826 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:58.213219 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:58.213318 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:58.217667 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:58.217684 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:58.217700 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:53:58.217747 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:58.220443 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:58.221053 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:58.221059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:58.221061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:58.221211 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:53:58.221618 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:53:58.221621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:53:58.221623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:53:58.221959 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:58.222165 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:58.222492 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 02:53:58.222659 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:53:58.223840 23590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:58.310235 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 23467
I20260929 02:53:58.323211 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 23107
I20260929 02:53:58.324251 23169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:53:58.433408 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 23107
2026-09-29T02:53:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4108 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T02:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:53:58Z Disabled control of system clock
I20260929 02:53:58.502825 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36437
--webserver_interface=127.4.197.190
--webserver_port=0
--builtin_ntp_servers=127.4.197.148:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:36437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:53:58.824683 23648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:58.824958 23648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:58.825080 23648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:58.832080 23648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:53:58.832172 23648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:58.832242 23648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:58.832319 23648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:53:58.832379 23648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:53:58.841249 23648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:36437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:58.842442 23648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:58.844266 23648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:58.855683 23653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:58.855633 23656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:58.855713 23654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:58.856891 23648 server_base.cc:1034] running on GCE node
I20260929 02:53:58.857738 23648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:58.859251 23648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:58.860483 23648 hybrid_clock.cc:648] HybridClock initialized: now 1790650438860423 us; error 47 us; skew 500 ppm
I20260929 02:53:58.860855 23648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:53:58.863176 23648 webserver.cc:460] Webserver started at http://127.4.197.190:35071/ using document root <none> and password file <none>
I20260929 02:53:58.863848 23648 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:58.863958 23648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:58.864256 23648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:58.867230 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "94147bbd0b9148cb979f58349a2434e7"
format_stamp: "Formatted at 2026-09-29 02:53:58 on dist-test-slave-t6c1"
server_key: "1252fad79e14a655d49dc7e088119236"
server_key_iv: "b91f66517212423944a9b34712e8b458"
server_key_version: "encryptionkey@0"
I20260929 02:53:58.868007 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "94147bbd0b9148cb979f58349a2434e7"
format_stamp: "Formatted at 2026-09-29 02:53:58 on dist-test-slave-t6c1"
server_key: "1252fad79e14a655d49dc7e088119236"
server_key_iv: "b91f66517212423944a9b34712e8b458"
server_key_version: "encryptionkey@0"
I20260929 02:53:58.873039 23648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:53:58.876488 23662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:58.877820 23648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:53:58.878065 23648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "94147bbd0b9148cb979f58349a2434e7"
format_stamp: "Formatted at 2026-09-29 02:53:58 on dist-test-slave-t6c1"
server_key: "1252fad79e14a655d49dc7e088119236"
server_key_iv: "b91f66517212423944a9b34712e8b458"
server_key_version: "encryptionkey@0"
I20260929 02:53:58.878312 23648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:58.895162 23648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:58.896003 23648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:58.896263 23648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:58.912994 23648 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:36437
I20260929 02:53:58.913012 23713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:36437 every 8 connection(s)
I20260929 02:53:58.914361 23648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:53:58.918021 23714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:53:58.923261 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 23648
I20260929 02:53:58.923435 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:53:58.923972 4886 external_mini_cluster.cc:1428] Setting key 3878d0fdb43e8c7ffeb7edcaa23bb81c
I20260929 02:53:58.929246 23714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7: Bootstrap starting.
I20260929 02:53:58.932397 23714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7: Neither blocks nor log segments found. Creating new log.
I20260929 02:53:58.933442 23714 log.cc:824] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:53:58.936655 23714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7: No bootstrap required, opened a new log
I20260929 02:53:58.937539 23717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:58.925913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.4.197.190:36437)
0929 02:53:58.927047 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:58.927062 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:58.927104 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:53:58.928823 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:58.928848 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:58.929322 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:58.929740 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:58.930290 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:58.931652 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:58.932995 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:58.933732 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:58.933895 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:58.935160 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:58.935169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:58.935184 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:58.935741 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:58.935826 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:58.935850 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:58.935945 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:58.936180 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 02:53:58.936549 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":385,"threads_started":1}
I20260929 02:53:58.944624 23714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94147bbd0b9148cb979f58349a2434e7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36437 } }
I20260929 02:53:58.945060 23714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:53:58.945180 23714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94147bbd0b9148cb979f58349a2434e7, State: Initialized, Role: FOLLOWER
I20260929 02:53:58.945782 23714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94147bbd0b9148cb979f58349a2434e7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36437 } }
I20260929 02:53:58.946048 23714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:53:58.946168 23714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:53:58.946316 23714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:53:58.948549 23714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94147bbd0b9148cb979f58349a2434e7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36437 } }
I20260929 02:53:58.949039 23714 leader_election.cc:302] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94147bbd0b9148cb979f58349a2434e7; no voters:
I20260929 02:53:58.949595 23714 leader_election.cc:288] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:53:58.949746 23719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:53:58.950407 23719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [term 1 LEADER]: Becoming Leader. State: Replica: 94147bbd0b9148cb979f58349a2434e7, State: Running, Role: LEADER
I20260929 02:53:58.950963 23719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94147bbd0b9148cb979f58349a2434e7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36437 } }
I20260929 02:53:58.951733 23714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:53:58.953981 23721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94147bbd0b9148cb979f58349a2434e7. Latest consensus state: current_term: 1 leader_uuid: "94147bbd0b9148cb979f58349a2434e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94147bbd0b9148cb979f58349a2434e7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36437 } } }
I20260929 02:53:58.954152 23720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94147bbd0b9148cb979f58349a2434e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94147bbd0b9148cb979f58349a2434e7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 36437 } } }
I20260929 02:53:58.954388 23721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:58.954450 23720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:53:58.956117 23728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:53:58.960376 23728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:53:58.969308 23728 catalog_manager.cc:1347] Generated new cluster ID: 2715bff83e5040579121ff98ba614db9
I20260929 02:53:58.969424 23728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:53:58.996276 23728 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:53:58.998145 23728 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:53:59.009666 23728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7: Generated new TSK 0
I20260929 02:53:59.010682 23728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:53:59.017287 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36437
--builtin_ntp_servers=127.4.197.148:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:53:59.336925 23738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:59.337212 23738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:59.337296 23738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:59.344585 23738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:59.344702 23738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:59.344810 23738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:53:59.354310 23738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:59.355787 23738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:59.357759 23738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:59.371403 23744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:59.371423 23746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:59.371451 23743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:59.372280 23738 server_base.cc:1034] running on GCE node
I20260929 02:53:59.372876 23738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:59.373803 23738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:59.375021 23738 hybrid_clock.cc:648] HybridClock initialized: now 1790650439374971 us; error 36 us; skew 500 ppm
I20260929 02:53:59.375334 23738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:53:59.377524 23738 webserver.cc:460] Webserver started at http://127.4.197.129:43517/ using document root <none> and password file <none>
I20260929 02:53:59.378185 23738 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:59.378278 23738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:59.378607 23738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:59.381444 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "3c8654ca6ee240e89fb0c70db228bbbe"
format_stamp: "Formatted at 2026-09-29 02:53:59 on dist-test-slave-t6c1"
server_key: "ec03c86f9e757041cd3f7c9c2a89a33b"
server_key_iv: "0722f6c5e1ec36bfa09f45eceeacb89c"
server_key_version: "encryptionkey@0"
I20260929 02:53:59.382207 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "3c8654ca6ee240e89fb0c70db228bbbe"
format_stamp: "Formatted at 2026-09-29 02:53:59 on dist-test-slave-t6c1"
server_key: "ec03c86f9e757041cd3f7c9c2a89a33b"
server_key_iv: "0722f6c5e1ec36bfa09f45eceeacb89c"
server_key_version: "encryptionkey@0"
I20260929 02:53:59.387192 23738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:53:59.390401 23752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:59.391776 23738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:53:59.391932 23738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "3c8654ca6ee240e89fb0c70db228bbbe"
format_stamp: "Formatted at 2026-09-29 02:53:59 on dist-test-slave-t6c1"
server_key: "ec03c86f9e757041cd3f7c9c2a89a33b"
server_key_iv: "0722f6c5e1ec36bfa09f45eceeacb89c"
server_key_version: "encryptionkey@0"
I20260929 02:53:59.392136 23738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:59.423854 23738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:59.424719 23738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:59.424995 23738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:59.425972 23738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:59.427433 23738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:59.427533 23738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:59.427639 23738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:59.427709 23738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:59.458240 23738 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:43423
I20260929 02:53:59.458288 23864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:43423 every 8 connection(s)
I20260929 02:53:59.459625 23738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:53:59.470001 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 23738
I20260929 02:53:59.470153 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:53:59.470867 4886 external_mini_cluster.cc:1428] Setting key c629e245b45f5a6be71556b600a38911
I20260929 02:53:59.471657 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:59.462041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:57479 (local address 127.4.197.190:36437)
0929 02:53:59.462522 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:59.462528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:59.463778 (+ 1250us) server_negotiation.cc:407] Connection header received
0929 02:53:59.465581 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:59.465585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:59.465660 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:59.465747 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:59.466813 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:59.467600 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:59.469128 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:59.469528 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:59.469638 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:59.470872 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:59.470878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:59.470883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:59.470906 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:59.470995 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:59.471013 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:59.471105 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:59.471281 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 02:53:59.471390 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":213,"threads_started":1}
I20260929 02:53:59.472681 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:59.462696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36437 (local address 127.4.197.129:57479)
0929 02:53:59.463620 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:59.463658 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:59.465308 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:59.465974 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:59.465984 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:59.466083 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:59.466685 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:59.466701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:59.467739 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:59.467743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:59.469021 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:59.469034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:59.469726 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:59.469731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:59.469936 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:59.470541 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:53:59.470544 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:59.470588 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:53:59.470667 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:59.471100 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:59.471109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:59.471114 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:53:59.471245 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 02:53:59.471585 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":279,"threads_started":1}
I20260929 02:53:59.473591 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36437
--builtin_ntp_servers=127.4.197.148:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:53:59.474104 23865 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36437
I20260929 02:53:59.474370 23865 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:59.475095 23865 heartbeater.cc:511] Master 127.4.197.190:36437 requested a full tablet report, sending...
I20260929 02:53:59.476948 23679 ts_manager.cc:194] Registered new tserver with Master: 3c8654ca6ee240e89fb0c70db228bbbe (127.4.197.129:43423)
I20260929 02:53:59.478410 23679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.129:57479
W20260929 02:53:59.790887 23869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:53:59.791213 23869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:53:59.791316 23869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:53:59.798692 23869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:53:59.798784 23869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:53:59.798893 23869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:53:59.807792 23869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:53:59.808950 23869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:53:59.810599 23869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:53:59.822407 23875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:59.822428 23877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:53:59.822863 23874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:53:59.823251 23869 server_base.cc:1034] running on GCE node
I20260929 02:53:59.823776 23869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:53:59.824644 23869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:53:59.825847 23869 hybrid_clock.cc:648] HybridClock initialized: now 1790650439825791 us; error 52 us; skew 500 ppm
I20260929 02:53:59.826138 23869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:53:59.828223 23869 webserver.cc:460] Webserver started at http://127.4.197.130:45045/ using document root <none> and password file <none>
I20260929 02:53:59.828889 23869 fs_manager.cc:359] Metadata directory not provided
I20260929 02:53:59.828989 23869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:53:59.829288 23869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:53:59.832140 23869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "6a00c800030740bda5ff66177c09c509"
format_stamp: "Formatted at 2026-09-29 02:53:59 on dist-test-slave-t6c1"
server_key: "8cfb348d44f66c1ba07372c588e46399"
server_key_iv: "d2480bd9df30ce675ea600cc620c0453"
server_key_version: "encryptionkey@0"
I20260929 02:53:59.832921 23869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "6a00c800030740bda5ff66177c09c509"
format_stamp: "Formatted at 2026-09-29 02:53:59 on dist-test-slave-t6c1"
server_key: "8cfb348d44f66c1ba07372c588e46399"
server_key_iv: "d2480bd9df30ce675ea600cc620c0453"
server_key_version: "encryptionkey@0"
I20260929 02:53:59.838017 23869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:53:59.841466 23883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:59.842798 23869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:53:59.842994 23869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "6a00c800030740bda5ff66177c09c509"
format_stamp: "Formatted at 2026-09-29 02:53:59 on dist-test-slave-t6c1"
server_key: "8cfb348d44f66c1ba07372c588e46399"
server_key_iv: "d2480bd9df30ce675ea600cc620c0453"
server_key_version: "encryptionkey@0"
I20260929 02:53:59.843194 23869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:53:59.856307 23869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:53:59.857102 23869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:53:59.857366 23869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:53:59.858289 23869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:53:59.859828 23869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:53:59.859926 23869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:59.860033 23869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:53:59.860097 23869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:53:59.888079 23869 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:45877
I20260929 02:53:59.888118 23995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:45877 every 8 connection(s)
I20260929 02:53:59.889389 23869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:53:59.894400 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 23869
I20260929 02:53:59.894577 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:53:59.895200 4886 external_mini_cluster.cc:1428] Setting key a6d11ea76edc46318a5958efa2ce49b3
I20260929 02:53:59.897926 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36437
--builtin_ntp_servers=127.4.197.148:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:53:59.902781 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:59.891957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:42473 (local address 127.4.197.190:36437)
0929 02:53:59.892173 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:53:59.892178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:53:59.893565 (+ 1387us) server_negotiation.cc:407] Connection header received
0929 02:53:59.895904 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:53:59.895910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:53:59.895992 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:53:59.896087 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:53:59.897189 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:59.897878 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:59.900055 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:53:59.900531 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:53:59.900628 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:59.902085 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:53:59.902090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:53:59.902093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:53:59.902112 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:53:59.902147 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:53:59.902158 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:53:59.902228 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:53:59.902450 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 02:53:59.902541 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 02:53:59.904217 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:53:59.892552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36437 (local address 127.4.197.130:42473)
0929 02:53:59.893415 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0929 02:53:59.893456 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:53:59.895577 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:53:59.896364 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:53:59.896377 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:53:59.896473 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:53:59.897035 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:59.897055 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:59.898030 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:59.898035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:59.899899 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:53:59.899912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:53:59.900731 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:53:59.900737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:53:59.901022 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:53:59.901732 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:53:59.901735 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:53:59.901791 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:53:59.901888 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:53:59.902238 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:53:59.902247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:53:59.902256 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:53:59.902433 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 02:53:59.902848 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":250,"threads_started":1}
I20260929 02:53:59.905553 23996 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36437
I20260929 02:53:59.905854 23996 heartbeater.cc:464] Registering TS with master...
I20260929 02:53:59.906718 23996 heartbeater.cc:511] Master 127.4.197.190:36437 requested a full tablet report, sending...
I20260929 02:53:59.908207 23679 ts_manager.cc:194] Registered new tserver with Master: 6a00c800030740bda5ff66177c09c509 (127.4.197.130:45877)
I20260929 02:53:59.908996 23679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.130:42473
W20260929 02:54:00.215307 23999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:00.215588 23999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:00.215719 23999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:00.223210 23999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:00.223313 23999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:00.223461 23999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:54:00.232621 23999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:00.233963 23999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:00.235833 23999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:00.248780 24005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:00.248734 24004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:00.248745 24007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:00.249655 23999 server_base.cc:1034] running on GCE node
I20260929 02:54:00.250274 23999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:00.251286 23999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:00.252493 23999 hybrid_clock.cc:648] HybridClock initialized: now 1790650440252425 us; error 42 us; skew 500 ppm
I20260929 02:54:00.252846 23999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:54:00.255079 23999 webserver.cc:460] Webserver started at http://127.4.197.131:34275/ using document root <none> and password file <none>
I20260929 02:54:00.255755 23999 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:00.255841 23999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:00.256140 23999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:00.259055 23999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "d1e451c660df4f1bb8f1e5561a0ea859"
format_stamp: "Formatted at 2026-09-29 02:54:00 on dist-test-slave-t6c1"
server_key: "1a27e4d58252026ea518a0915ac70e6f"
server_key_iv: "3c3c5bc2146ad7f9a2a4a0287ba88882"
server_key_version: "encryptionkey@0"
I20260929 02:54:00.259812 23999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "d1e451c660df4f1bb8f1e5561a0ea859"
format_stamp: "Formatted at 2026-09-29 02:54:00 on dist-test-slave-t6c1"
server_key: "1a27e4d58252026ea518a0915ac70e6f"
server_key_iv: "3c3c5bc2146ad7f9a2a4a0287ba88882"
server_key_version: "encryptionkey@0"
I20260929 02:54:00.264950 23999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260929 02:54:00.268445 24013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:00.269820 23999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 02:54:00.269976 23999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "d1e451c660df4f1bb8f1e5561a0ea859"
format_stamp: "Formatted at 2026-09-29 02:54:00 on dist-test-slave-t6c1"
server_key: "1a27e4d58252026ea518a0915ac70e6f"
server_key_iv: "3c3c5bc2146ad7f9a2a4a0287ba88882"
server_key_version: "encryptionkey@0"
I20260929 02:54:00.270179 23999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:00.294013 23999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:00.294847 23999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:00.295163 23999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:00.296125 23999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:54:00.297557 23999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:54:00.297649 23999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:00.297751 23999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:54:00.297821 23999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:00.326371 23999 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:39807
I20260929 02:54:00.326436 24125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:39807 every 8 connection(s)
I20260929 02:54:00.327669 23999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:54:00.333825 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 23999
I20260929 02:54:00.334013 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:54:00.334540 4886 external_mini_cluster.cc:1428] Setting key 300dceffa87828448f328abb70ed2445
I20260929 02:54:00.340473 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.330148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:40589 (local address 127.4.197.190:36437)
0929 02:54:00.330387 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.330393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.331810 (+ 1417us) server_negotiation.cc:407] Connection header received
0929 02:54:00.333595 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.333599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.333669 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.333755 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:00.335241 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.336172 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.338004 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.338423 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.338519 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.339823 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:00.339827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:00.339830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:00.339849 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:00.339889 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:00.339901 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:00.339968 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.340152 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.340235 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:54:00.341573 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.330728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:36437 (local address 127.4.197.131:40589)
0929 02:54:00.331661 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:00.331697 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:00.333318 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:00.334049 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:00.334062 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:00.334200 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:00.335054 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.335076 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.336350 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.336356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.337874 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.337887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.338602 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.338607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.338824 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.339516 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:54:00.339518 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:00.339560 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:54:00.339651 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:00.339972 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:00.339978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:00.339984 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:54:00.340131 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 02:54:00.340466 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":297,"threads_started":1}
I20260929 02:54:00.342960 24126 heartbeater.cc:347] Connected to a master server at 127.4.197.190:36437
I20260929 02:54:00.343235 24126 heartbeater.cc:464] Registering TS with master...
I20260929 02:54:00.343953 24126 heartbeater.cc:511] Master 127.4.197.190:36437 requested a full tablet report, sending...
I20260929 02:54:00.345263 23679 ts_manager.cc:194] Registered new tserver with Master: d1e451c660df4f1bb8f1e5561a0ea859 (127.4.197.131:39807)
I20260929 02:54:00.346155 23679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.131:40589
I20260929 02:54:00.349561 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:54:00.358753 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.353422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.4.197.190:36437)
0929 02:54:00.353613 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.353621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.353976 (+ 355us) server_negotiation.cc:407] Connection header received
0929 02:54:00.354102 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.354105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.354177 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.354256 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:00.354658 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.355453 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.356744 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.357126 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.357209 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.358127 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:00.358131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:00.358134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:00.358151 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:00.358181 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:00.358192 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:00.358294 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.358422 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.358484 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:54:00.362229 23679 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:54:00.368839 23679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:54:00.397073 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.380754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:45877 (local address 127.0.0.1:39168)
0929 02:54:00.388206 (+ 7452us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:00.388251 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:00.388442 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:00.389864 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:00.389872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:00.389913 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:00.390368 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.390382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.392105 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.392111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.394433 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.394448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.396316 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.396322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.396509 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.396563 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 02:54:00.396689 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 02:54:00.396822 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6692,"thread_start_us":6516,"threads_started":1}
I20260929 02:54:00.398075 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.380864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.4.197.129:43423)
0929 02:54:00.390634 (+ 9770us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.390640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.390660 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:54:00.390739 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.390744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.390990 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.391843 (+ 853us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:00.392079 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.393516 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.395734 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.397066 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.397195 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.397416 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.397694 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.397816 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8837,"thread_start_us":8893,"threads_started":1}
I20260929 02:54:00.398093 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.380754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43423 (local address 127.0.0.1:35384)
0929 02:54:00.388451 (+ 7697us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:00.388471 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:00.388600 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:00.391168 (+ 2568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:00.391172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:00.391199 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:00.391644 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.391655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.393699 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.393708 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.395531 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.395545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.397341 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.397348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.397526 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.397557 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:54:00.397639 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 02:54:00.397754 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7541,"mutex_wait_us":338,"spinlock_wait_cycles":768,"thread_start_us":8975,"threads_started":1}
I20260929 02:54:00.398397 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.381363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.4.197.130:45877)
0929 02:54:00.389191 (+ 7828us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.389201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.389236 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:54:00.389353 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.389363 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.389645 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.389872 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:00.390566 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.391934 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.394631 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.396112 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.397554 (+ 1442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.397818 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.397900 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.398051 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7581,"thread_start_us":7433,"threads_started":1}
I20260929 02:54:00.399061 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.381265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:39807 (local address 127.0.0.1:43496)
0929 02:54:00.390357 (+ 9092us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:00.390374 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:00.390504 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:00.391135 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:00.391140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:00.391165 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:00.391667 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.391680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.393531 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.393535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.395328 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.395342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.398312 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.398318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.398483 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.398516 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:54:00.398598 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 02:54:00.398717 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8966,"mutex_wait_us":1,"spinlock_wait_cycles":9344,"thread_start_us":6870,"threads_started":1}
I20260929 02:54:00.399775 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.381894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.4.197.131:39807)
0929 02:54:00.389469 (+ 7575us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.389477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.390490 (+ 1013us) server_negotiation.cc:407] Connection header received
0929 02:54:00.390646 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.390654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.390911 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.391116 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:00.391860 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.393414 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.395520 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.397066 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.397385 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.397596 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.399316 (+ 1720us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.399477 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7427,"thread_start_us":7285,"threads_started":1}
I20260929 02:54:00.401937 23931 tablet_service.cc:1467] Processing CreateTablet for tablet 5209c47a6f024c5f8169a59a5a2e9f66 (DEFAULT_TABLE table=test-table [id=dbc9da7fb89f46ae8666a5079c7e86c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:54:00.401937 23800 tablet_service.cc:1467] Processing CreateTablet for tablet 5209c47a6f024c5f8169a59a5a2e9f66 (DEFAULT_TABLE table=test-table [id=dbc9da7fb89f46ae8666a5079c7e86c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:54:00.402657 24061 tablet_service.cc:1467] Processing CreateTablet for tablet 5209c47a6f024c5f8169a59a5a2e9f66 (DEFAULT_TABLE table=test-table [id=dbc9da7fb89f46ae8666a5079c7e86c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:54:00.403386 23800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5209c47a6f024c5f8169a59a5a2e9f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:00.403393 23931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5209c47a6f024c5f8169a59a5a2e9f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:00.403766 24061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5209c47a6f024c5f8169a59a5a2e9f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:00.413897 24144 tablet_bootstrap.cc:492] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: Bootstrap starting.
I20260929 02:54:00.416996 24144 tablet_bootstrap.cc:654] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:00.417119 24145 tablet_bootstrap.cc:492] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: Bootstrap starting.
I20260929 02:54:00.418675 24144 log.cc:824] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:00.420032 24146 tablet_bootstrap.cc:492] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: Bootstrap starting.
I20260929 02:54:00.421162 24145 tablet_bootstrap.cc:654] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:00.421481 24144 tablet_bootstrap.cc:492] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: No bootstrap required, opened a new log
I20260929 02:54:00.421664 24144 ts_tablet_manager.cc:1397] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 02:54:00.422289 24145 log.cc:824] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:00.423271 24146 tablet_bootstrap.cc:654] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:00.424336 24146 log.cc:824] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:00.425174 24145 tablet_bootstrap.cc:492] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: No bootstrap required, opened a new log
I20260929 02:54:00.425400 24145 ts_tablet_manager.cc:1397] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 02:54:00.427117 24146 tablet_bootstrap.cc:492] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: No bootstrap required, opened a new log
I20260929 02:54:00.427358 24146 ts_tablet_manager.cc:1397] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:54:00.430472 24144 raft_consensus.cc:348] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 } } peers { permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 } } peers { permanent_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39807 } }
I20260929 02:54:00.430940 24144 raft_consensus.cc:374] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:00.431038 24144 raft_consensus.cc:729] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e451c660df4f1bb8f1e5561a0ea859, State: Initialized, Role: FOLLOWER
I20260929 02:54:00.431656 24144 consensus_queue.cc:260] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 } } peers { permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 } } peers { permanent_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39807 } }
I20260929 02:54:00.432742 24126 heartbeater.cc:503] Master 127.4.197.190:36437 was elected leader, sending a full tablet report...
I20260929 02:54:00.432672 24145 raft_consensus.cc:348] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 } } peers { permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 } } peers { permanent_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39807 } }
I20260929 02:54:00.433069 24145 raft_consensus.cc:374] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:00.433156 24145 raft_consensus.cc:729] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a00c800030740bda5ff66177c09c509, State: Initialized, Role: FOLLOWER
I20260929 02:54:00.433408 24144 ts_tablet_manager.cc:1428] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260929 02:54:00.433674 24145 consensus_queue.cc:260] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 } } peers { permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 } } peers { permanent_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39807 } }
I20260929 02:54:00.434727 23996 heartbeater.cc:503] Master 127.4.197.190:36437 was elected leader, sending a full tablet report...
I20260929 02:54:00.435348 24145 ts_tablet_manager.cc:1428] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260929 02:54:00.436765 24146 raft_consensus.cc:348] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 } } peers { permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 } } peers { permanent_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39807 } }
I20260929 02:54:00.437201 24146 raft_consensus.cc:374] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:00.437287 24146 raft_consensus.cc:729] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8654ca6ee240e89fb0c70db228bbbe, State: Initialized, Role: FOLLOWER
I20260929 02:54:00.437852 24146 consensus_queue.cc:260] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 } } peers { permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 } } peers { permanent_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39807 } }
I20260929 02:54:00.439049 23865 heartbeater.cc:503] Master 127.4.197.190:36437 was elected leader, sending a full tablet report...
I20260929 02:54:00.439611 24146 ts_tablet_manager.cc:1428] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260929 02:54:00.440084 24150 raft_consensus.cc:482] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:54:00.440375 24150 raft_consensus.cc:504] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 } } peers { permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 } } peers { permanent_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39807 } }
I20260929 02:54:00.442077 24150 leader_election.cc:288] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c8654ca6ee240e89fb0c70db228bbbe (127.4.197.129:43423), 6a00c800030740bda5ff66177c09c509 (127.4.197.130:45877)
I20260929 02:54:00.449065 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.442711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:45877 (local address 127.4.197.131:35633)
0929 02:54:00.443732 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:00.443751 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:00.443870 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:00.444340 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:00.444352 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:00.444385 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:00.444576 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.444586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.445826 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.445831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.447213 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.447225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.448390 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.448396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.448561 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.448584 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:54:00.448660 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 02:54:00.448793 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902}
I20260929 02:54:00.449071 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.442889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:35633 (local address 127.4.197.130:45877)
0929 02:54:00.443114 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.443120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.443999 (+ 879us) server_negotiation.cc:407] Connection header received
0929 02:54:00.444071 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.444077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.444143 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.444230 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:00.444719 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.445726 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.447363 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.448244 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.448381 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.448450 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.448697 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.448816 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 02:54:00.450017 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.443714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:43423 (local address 127.4.197.131:51979)
0929 02:54:00.444624 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:00.444640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:00.444806 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:00.445248 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:00.445253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:00.445276 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:00.445627 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.445637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.446881 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.446885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.448292 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:00.448305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.449371 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.449377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:00.449533 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.449584 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 02:54:00.449668 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 02:54:00.449759 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":576,"threads_started":1}
I20260929 02:54:00.450138 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.443890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:51979 (local address 127.4.197.129:43423)
0929 02:54:00.444063 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.444068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.444713 (+ 645us) server_negotiation.cc:407] Connection header received
0929 02:54:00.444945 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.444950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.445025 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.445084 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:00.445791 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.446739 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.448468 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.449230 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.449384 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.449446 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.449737 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.449895 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:54:00.450140 23951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5209c47a6f024c5f8169a59a5a2e9f66" candidate_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a00c800030740bda5ff66177c09c509" is_pre_election: true
I20260929 02:54:00.450763 23951 raft_consensus.cc:2454] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e451c660df4f1bb8f1e5561a0ea859 in term 0.
I20260929 02:54:00.451050 23820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5209c47a6f024c5f8169a59a5a2e9f66" candidate_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" is_pre_election: true
I20260929 02:54:00.451453 24017 leader_election.cc:302] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a00c800030740bda5ff66177c09c509, d1e451c660df4f1bb8f1e5561a0ea859; no voters:
I20260929 02:54:00.451592 23820 raft_consensus.cc:2454] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e451c660df4f1bb8f1e5561a0ea859 in term 0.
I20260929 02:54:00.451821 24152 raft_consensus.cc:2781] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:54:00.451979 24152 raft_consensus.cc:482] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:54:00.452122 24152 raft_consensus.cc:3037] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:00.454423 24152 raft_consensus.cc:504] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 } } peers { permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 } } peers { permanent_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39807 } }
I20260929 02:54:00.455087 24152 leader_election.cc:288] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [CANDIDATE]: Term 1 election: Requested vote from peers 3c8654ca6ee240e89fb0c70db228bbbe (127.4.197.129:43423), 6a00c800030740bda5ff66177c09c509 (127.4.197.130:45877)
I20260929 02:54:00.455535 23951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5209c47a6f024c5f8169a59a5a2e9f66" candidate_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a00c800030740bda5ff66177c09c509"
I20260929 02:54:00.455555 23820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5209c47a6f024c5f8169a59a5a2e9f66" candidate_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8654ca6ee240e89fb0c70db228bbbe"
I20260929 02:54:00.455740 23820 raft_consensus.cc:3037] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:00.455717 23951 raft_consensus.cc:3037] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:00.458045 23820 raft_consensus.cc:2454] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e451c660df4f1bb8f1e5561a0ea859 in term 1.
I20260929 02:54:00.458168 23951 raft_consensus.cc:2454] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e451c660df4f1bb8f1e5561a0ea859 in term 1.
I20260929 02:54:00.458506 24014 leader_election.cc:302] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8654ca6ee240e89fb0c70db228bbbe, d1e451c660df4f1bb8f1e5561a0ea859; no voters:
I20260929 02:54:00.458808 24152 raft_consensus.cc:2781] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:00.459020 24152 raft_consensus.cc:686] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 1 LEADER]: Becoming Leader. State: Replica: d1e451c660df4f1bb8f1e5561a0ea859, State: Running, Role: LEADER
I20260929 02:54:00.459518 24152 consensus_queue.cc:237] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 } } peers { permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 } } peers { permanent_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39807 } }
W20260929 02:54:00.461660 23866 tablet.cc:2369] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:54:00.464529 23678 catalog_manager.cc:5495] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e451c660df4f1bb8f1e5561a0ea859 (127.4.197.131). New cstate: current_term: 1 leader_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e451c660df4f1bb8f1e5561a0ea859" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 39807 } health_report { overall_health: HEALTHY } } }
I20260929 02:54:00.498617 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.493699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.4.197.131:39807)
0929 02:54:00.493866 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.493870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.493898 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:54:00.494038 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.494042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.494124 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.494248 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:00.494590 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.495356 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.496749 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.497155 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.497243 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.497705 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:00.498183 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:00.498309 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.498396 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.498432 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:54:00.506210 23820 raft_consensus.cc:1261] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe [term 1 FOLLOWER]: Refusing update from remote peer d1e451c660df4f1bb8f1e5561a0ea859: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:54:00.506213 23951 raft_consensus.cc:1261] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509 [term 1 FOLLOWER]: Refusing update from remote peer d1e451c660df4f1bb8f1e5561a0ea859: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:54:00.507083 24150 consensus_queue.cc:1035] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8654ca6ee240e89fb0c70db228bbbe" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:54:00.507339 24152 consensus_queue.cc:1035] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a00c800030740bda5ff66177c09c509" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:54:00.519475 24157 mvcc.cc:205] Tried to move back new op lower bound from 7334504204304572416 to 7334504204126437376. Current Snapshot: MvccSnapshot[applied={T|T < 7334504204304572416}]
I20260929 02:54:00.530675 23679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:54:00.530958 23679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:54:00.534514 23679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94147bbd0b9148cb979f58349a2434e7: Sending DeleteTablet for 3 replicas of tablet 5209c47a6f024c5f8169a59a5a2e9f66
I20260929 02:54:00.535737 23800 tablet_service.cc:1514] Processing DeleteTablet for tablet 5209c47a6f024c5f8169a59a5a2e9f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:54:00 UTC) from {username='slave'} at 127.0.0.1:35384
I20260929 02:54:00.535737 23931 tablet_service.cc:1514] Processing DeleteTablet for tablet 5209c47a6f024c5f8169a59a5a2e9f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:54:00 UTC) from {username='slave'} at 127.0.0.1:39168
I20260929 02:54:00.535754 24061 tablet_service.cc:1514] Processing DeleteTablet for tablet 5209c47a6f024c5f8169a59a5a2e9f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:54:00 UTC) from {username='slave'} at 127.0.0.1:43496
I20260929 02:54:00.539572 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 23738
I20260929 02:54:00.551429 24169 tablet_replica.cc:330] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: stopping tablet replica
I20260929 02:54:00.551903 24169 raft_consensus.cc:2229] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:54:00.552438 24167 tablet_replica.cc:330] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: stopping tablet replica
I20260929 02:54:00.552672 24169 raft_consensus.cc:2258] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:54:00.552991 24167 raft_consensus.cc:2229] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:54:00.553602 24167 raft_consensus.cc:2258] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:54:00.554368 24168 tablet_replica.cc:330] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: stopping tablet replica
I20260929 02:54:00.554792 24168 raft_consensus.cc:2229] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:54:00.555052 24169 ts_tablet_manager.cc:1905] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:54:00.555295 24168 raft_consensus.cc:2258] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:54:00.557358 24168 ts_tablet_manager.cc:1905] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:54:00.557679 24167 ts_tablet_manager.cc:1905] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:54:00.562786 24169 ts_tablet_manager.cc:1918] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:54:00.563033 24169 log.cc:1196] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/5209c47a6f024c5f8169a59a5a2e9f66
I20260929 02:54:00.563567 24169 ts_tablet_manager.cc:1939] T 5209c47a6f024c5f8169a59a5a2e9f66 P d1e451c660df4f1bb8f1e5561a0ea859: Deleting consensus metadata
I20260929 02:54:00.564586 24168 ts_tablet_manager.cc:1918] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:54:00.564816 24168 log.cc:1196] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/5209c47a6f024c5f8169a59a5a2e9f66
I20260929 02:54:00.565416 24168 ts_tablet_manager.cc:1939] T 5209c47a6f024c5f8169a59a5a2e9f66 P 6a00c800030740bda5ff66177c09c509: Deleting consensus metadata
I20260929 02:54:00.565461 23664 catalog_manager.cc:4841] TS d1e451c660df4f1bb8f1e5561a0ea859 (127.4.197.131:39807): tablet 5209c47a6f024c5f8169a59a5a2e9f66 (table test-table [id=dbc9da7fb89f46ae8666a5079c7e86c5]) successfully deleted
I20260929 02:54:00.565968 24167 ts_tablet_manager.cc:1918] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:54:00.566135 24167 log.cc:1196] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650137986910-4886-0/minicluster-data/ts-0/wal/wals/5209c47a6f024c5f8169a59a5a2e9f66
I20260929 02:54:00.566663 24167 ts_tablet_manager.cc:1939] T 5209c47a6f024c5f8169a59a5a2e9f66 P 3c8654ca6ee240e89fb0c70db228bbbe: Deleting consensus metadata
I20260929 02:54:00.567500 23666 catalog_manager.cc:4841] TS 6a00c800030740bda5ff66177c09c509 (127.4.197.130:45877): tablet 5209c47a6f024c5f8169a59a5a2e9f66 (table test-table [id=dbc9da7fb89f46ae8666a5079c7e86c5]) successfully deleted
I20260929 02:54:00.568140 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.540913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.4.197.129:43423)
0929 02:54:00.541094 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.541101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.551726 (+ 10625us) server_negotiation.cc:407] Connection header received
0929 02:54:00.551913 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.551920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.552048 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.552168 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:00.552990 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.554285 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.559033 (+ 4748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.559633 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.559733 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.566360 (+ 6627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:00.566372 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:00.566391 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:00.567188 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:00.567252 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:00.567279 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:00.567372 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.567632 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.567752 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:54:00.568307 23663 catalog_manager.cc:4841] TS 3c8654ca6ee240e89fb0c70db228bbbe (127.4.197.129:43423): tablet 5209c47a6f024c5f8169a59a5a2e9f66 (table test-table [id=dbc9da7fb89f46ae8666a5079c7e86c5]) successfully deleted
I20260929 02:54:00.569209 23860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:00.659432 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 23738
I20260929 02:54:00.673939 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 23869
I20260929 02:54:00.681293 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.674754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.4.197.130:45877)
0929 02:54:00.674981 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.674986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.675100 (+ 114us) server_negotiation.cc:407] Connection header received
0929 02:54:00.675301 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.675306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.675407 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.675486 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:00.675920 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.676621 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.678418 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.678785 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.678877 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.679985 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:00.679994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:00.680008 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:00.680581 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:00.680628 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:00.680652 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:00.680738 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.680887 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.680978 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 02:54:00.681962 23991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:00.769819 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 23869
I20260929 02:54:00.783440 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 23999
I20260929 02:54:00.791098 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:00.784283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.4.197.131:39807)
0929 02:54:00.784457 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:00.784461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:00.784478 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:54:00.784652 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:00.784655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:00.784737 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:00.784823 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:00.785325 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.786175 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.788002 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:00.788421 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:00.788520 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:00.789614 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:00.789621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:00.789635 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:00.790263 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:00.790312 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:00.790341 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:00.790418 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:00.790595 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 02:54:00.790673 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:54:00.791832 24121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:00.910858 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 23999
I20260929 02:54:00.926434 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 23648
I20260929 02:54:00.927605 23709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:01.034989 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 23648
2026-09-29T02:54:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2600 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:01 dist-test-slave-t6c1 krb5kdc[24177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:01 dist-test-slave-t6c1 krb5kdc[24177](info): set up 2 sockets
Sep 29 02:54:01 dist-test-slave-t6c1 krb5kdc[24177](info): commencing operation
krb5kdc: starting...
W20260929 02:54:02.063033 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:02 dist-test-slave-t6c1 krb5kdc[24177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:54:02.294026 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:42847
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:42847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:02.614522 24190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:02.614780 24190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:02.614866 24190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:02.621817 24190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:02.621906 24190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:02.621958 24190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:02.622014 24190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:02.622056 24190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:02.631603 24190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:42847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:42847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:02.632899 24190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:02.634217 24190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:02.646041 24198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:02.646071 24195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:02.646121 24196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:02.646853 24190 server_base.cc:1034] running on GCE node
I20260929 02:54:02.647670 24190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:02.649070 24190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:02.650274 24190 hybrid_clock.cc:648] HybridClock initialized: now 1790650442650215 us; error 41 us; skew 500 ppm
I20260929 02:54:02.650596 24190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:02 dist-test-slave-t6c1 krb5kdc[24177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650442, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:02.657111 24190 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:54:02.658550 24190 webserver.cc:460] Webserver started at http://127.4.197.190:45961/ using document root <none> and password file <none>
I20260929 02:54:02.659252 24190 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:02.659366 24190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:02.659648 24190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:02.662504 24190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "a95fe50323454366973faf61ac8434a1"
format_stamp: "Formatted at 2026-09-29 02:54:02 on dist-test-slave-t6c1"
server_key: "ca3ac17c231e284ecb80fa594cbeda03"
server_key_iv: "f4032f6523f1e03213df34d48203c66a"
server_key_version: "encryptionkey@0"
I20260929 02:54:02.663255 24190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "a95fe50323454366973faf61ac8434a1"
format_stamp: "Formatted at 2026-09-29 02:54:02 on dist-test-slave-t6c1"
server_key: "ca3ac17c231e284ecb80fa594cbeda03"
server_key_iv: "f4032f6523f1e03213df34d48203c66a"
server_key_version: "encryptionkey@0"
I20260929 02:54:02.668048 24190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:54:02.671212 24205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:02.672495 24190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:54:02.672653 24190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "a95fe50323454366973faf61ac8434a1"
format_stamp: "Formatted at 2026-09-29 02:54:02 on dist-test-slave-t6c1"
server_key: "ca3ac17c231e284ecb80fa594cbeda03"
server_key_iv: "f4032f6523f1e03213df34d48203c66a"
server_key_version: "encryptionkey@0"
I20260929 02:54:02.672820 24190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:02.704674 24190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:02.708658 24190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:02.708911 24190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:02.726377 24190 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:42847
I20260929 02:54:02.726384 24256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:42847 every 8 connection(s)
I20260929 02:54:02.727897 24190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:02.731740 24257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:02.734426 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 24190
I20260929 02:54:02.734611 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:02.735203 4886 external_mini_cluster.cc:1428] Setting key e010eb5609340264e1aad0736694f029
I20260929 02:54:02.744129 24257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1: Bootstrap starting.
I20260929 02:54:02.747980 24257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:02.749383 24257 log.cc:824] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:54:02 dist-test-slave-t6c1 krb5kdc[24177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:02.752465 24257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1: No bootstrap required, opened a new log
I20260929 02:54:02.758127 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:02.737147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.4.197.190:42847)
0929 02:54:02.737931 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:02.737943 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:02.737984 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:54:02.739689 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:02.739713 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:02.740120 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:02.740466 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:02.742233 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:02.743650 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:02.745379 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:02.746067 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:02.746197 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:02.751132 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:02.751141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:02.751157 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:02.751192 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:02.754086 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:02.754754 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:02.754761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:02.754771 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:02.754984 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:02.755364 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:02.755369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:02.755371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:02.755920 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:02.756183 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:02.756666 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 02:54:02.757087 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":360,"threads_started":1}
I20260929 02:54:02.760776 24257 raft_consensus.cc:348] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95fe50323454366973faf61ac8434a1" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42847 } }
I20260929 02:54:02.761224 24257 raft_consensus.cc:374] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:02.761327 24257 raft_consensus.cc:729] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95fe50323454366973faf61ac8434a1, State: Initialized, Role: FOLLOWER
I20260929 02:54:02.761821 24257 consensus_queue.cc:260] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95fe50323454366973faf61ac8434a1" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42847 } }
I20260929 02:54:02.762030 24257 raft_consensus.cc:388] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:02.762121 24257 raft_consensus.cc:482] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:02.762246 24257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:02.764750 24257 raft_consensus.cc:504] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95fe50323454366973faf61ac8434a1" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42847 } }
I20260929 02:54:02.765214 24257 leader_election.cc:302] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a95fe50323454366973faf61ac8434a1; no voters:
I20260929 02:54:02.765736 24257 leader_election.cc:288] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:02.765861 24262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:02.766482 24262 raft_consensus.cc:686] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [term 1 LEADER]: Becoming Leader. State: Replica: a95fe50323454366973faf61ac8434a1, State: Running, Role: LEADER
I20260929 02:54:02.767041 24262 consensus_queue.cc:237] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95fe50323454366973faf61ac8434a1" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42847 } }
I20260929 02:54:02.767663 24257 sys_catalog.cc:564] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:02.769727 24263 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a95fe50323454366973faf61ac8434a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95fe50323454366973faf61ac8434a1" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42847 } } }
I20260929 02:54:02.770145 24263 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:02.769752 24264 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a95fe50323454366973faf61ac8434a1. Latest consensus state: current_term: 1 leader_uuid: "a95fe50323454366973faf61ac8434a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95fe50323454366973faf61ac8434a1" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 42847 } } }
I20260929 02:54:02.770473 24264 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:02.779459 24268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:02.784508 24268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:02.794454 24268 catalog_manager.cc:1347] Generated new cluster ID: ddf1a4edcbda449e97fbd58ee8966e8f
I20260929 02:54:02.794705 24268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:02.845328 24268 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:02.846706 24268 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:02.857079 24268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1: Generated new TSK 0
I20260929 02:54:02.857962 24268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.197.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
Entry for principal kudu/127.4.197.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab.
I20260929 02:54:02.930059 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42847
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:03.261216 24283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:03.261502 24283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:03.261595 24283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:03.268954 24283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:03.269086 24283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:03.269240 24283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:54:03.279690 24283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.129.keytab
--principal=kudu/127.4.197.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:03.281018 24283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:03.282317 24283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:03.294754 24291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:03.294811 24289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:03.294986 24288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:03.295701 24283 server_base.cc:1034] running on GCE node
I20260929 02:54:03.296267 24283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:03.297118 24283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:03.298311 24283 hybrid_clock.cc:648] HybridClock initialized: now 1790650443298257 us; error 50 us; skew 500 ppm
I20260929 02:54:03.298599 24283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:03 dist-test-slave-t6c1 krb5kdc[24177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650443, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:03.301578 24283 init.cc:377] Logged in from keytab as kudu/127.4.197.129@KRBTEST.COM (short username kudu)
I20260929 02:54:03.302950 24283 webserver.cc:460] Webserver started at http://127.4.197.129:36481/ using document root <none> and password file <none>
I20260929 02:54:03.303555 24283 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:03.303649 24283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:03.303893 24283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:03.306538 24283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "e25763214aa94c96a1a13cd9e3a63312"
format_stamp: "Formatted at 2026-09-29 02:54:03 on dist-test-slave-t6c1"
server_key: "a3ec757a5101f69fc59bdf45fd78d3e3"
server_key_iv: "cf4e7cf1aa2753b9a6b6a608a1adf19a"
server_key_version: "encryptionkey@0"
I20260929 02:54:03.307263 24283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "e25763214aa94c96a1a13cd9e3a63312"
format_stamp: "Formatted at 2026-09-29 02:54:03 on dist-test-slave-t6c1"
server_key: "a3ec757a5101f69fc59bdf45fd78d3e3"
server_key_iv: "cf4e7cf1aa2753b9a6b6a608a1adf19a"
server_key_version: "encryptionkey@0"
I20260929 02:54:03.311973 24283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:54:03.315222 24298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:03.316555 24283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:54:03.316684 24283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "e25763214aa94c96a1a13cd9e3a63312"
format_stamp: "Formatted at 2026-09-29 02:54:03 on dist-test-slave-t6c1"
server_key: "a3ec757a5101f69fc59bdf45fd78d3e3"
server_key_iv: "cf4e7cf1aa2753b9a6b6a608a1adf19a"
server_key_version: "encryptionkey@0"
I20260929 02:54:03.316838 24283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:03.347826 24283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:03.351984 24283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:03.352216 24283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:03.353132 24283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:54:03.354359 24283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:54:03.354439 24283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:03.354527 24283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:54:03.354576 24283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:03.384338 24283 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:45273
I20260929 02:54:03.384392 24410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:45273 every 8 connection(s)
I20260929 02:54:03.385658 24283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:54:03.387313 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 24283
I20260929 02:54:03.387468 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:54:03.387984 4886 external_mini_cluster.cc:1428] Setting key 89c65f507b2bdcb5efb1f56fd752f9c9
Sep 29 02:54:03 dist-test-slave-t6c1 krb5kdc[24177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650443, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.129@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:03.405591 24413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:03.388299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:46747 (local address 127.4.197.190:42847)
0929 02:54:03.388719 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:03.388725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:03.389899 (+ 1174us) server_negotiation.cc:407] Connection header received
0929 02:54:03.391867 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:03.391873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:03.391953 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:03.392058 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:03.393499 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:03.394454 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:03.396012 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:03.396563 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:03.396685 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:03.400130 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:03.400135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:03.400139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:03.400173 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:03.402550 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:03.403330 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:03.403334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:03.403337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:03.403457 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:03.403874 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:03.403877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:03.403879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:03.404141 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:03.404274 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:03.404994 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 02:54:03.405233 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":213,"threads_started":1}
I20260929 02:54:03.406580 24414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:03.388972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42847 (local address 127.4.197.129:46747)
0929 02:54:03.389741 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:03.389793 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:03.391604 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:03.392254 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:03.392262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:03.392781 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:03.393342 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:03.393359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:03.394584 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:03.394588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:03.395873 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:03.395895 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:03.396768 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:03.396772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:03.396980 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:03.397613 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:03.397636 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:03.399898 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:03.402690 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:03.402700 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:03.402718 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:03.403205 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:03.403559 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:03.403564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:03.403567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:03.403754 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:03.404264 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:03.404271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:03.404625 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 02:54:03.404866 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 02:54:03.405423 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":325,"threads_started":1}
I20260929 02:54:03.408281 24411 heartbeater.cc:347] Connected to a master server at 127.4.197.190:42847
I20260929 02:54:03.408568 24411 heartbeater.cc:464] Registering TS with master...
I20260929 02:54:03.409269 24411 heartbeater.cc:511] Master 127.4.197.190:42847 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.197.130@KRBTEST.COM; defaulting to no policy
I20260929 02:54:03.411382 24222 ts_manager.cc:194] Registered new tserver with Master: e25763214aa94c96a1a13cd9e3a63312 (127.4.197.129:45273)
I20260929 02:54:03.412950 24222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.129@KRBTEST.COM'} at 127.4.197.129:46747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
Entry for principal kudu/127.4.197.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab.
I20260929 02:54:03.439618 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42847
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:03.755898 24417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:03.756158 24417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:03.756289 24417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:03.763382 24417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:03.763499 24417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:03.763650 24417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:54:03.773034 24417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.130.keytab
--principal=kudu/127.4.197.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:03.774267 24417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:03.775564 24417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:03.788239 24423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:03.788260 24425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:03.788348 24422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:03.788916 24417 server_base.cc:1034] running on GCE node
I20260929 02:54:03.789454 24417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:03.790319 24417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:03.791507 24417 hybrid_clock.cc:648] HybridClock initialized: now 1790650443791464 us; error 37 us; skew 500 ppm
I20260929 02:54:03.791816 24417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:03 dist-test-slave-t6c1 krb5kdc[24177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650443, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:03.794597 24417 init.cc:377] Logged in from keytab as kudu/127.4.197.130@KRBTEST.COM (short username kudu)
I20260929 02:54:03.795958 24417 webserver.cc:460] Webserver started at http://127.4.197.130:39921/ using document root <none> and password file <none>
I20260929 02:54:03.796576 24417 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:03.796685 24417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:03.796969 24417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:03.799763 24417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "86245bed97d74d5daff4e50f9a768a72"
format_stamp: "Formatted at 2026-09-29 02:54:03 on dist-test-slave-t6c1"
server_key: "554f7f5d3f5d93ca11a82a61f78c57b8"
server_key_iv: "9af7f74787d46ca91629cc331164e9f2"
server_key_version: "encryptionkey@0"
I20260929 02:54:03.800516 24417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "86245bed97d74d5daff4e50f9a768a72"
format_stamp: "Formatted at 2026-09-29 02:54:03 on dist-test-slave-t6c1"
server_key: "554f7f5d3f5d93ca11a82a61f78c57b8"
server_key_iv: "9af7f74787d46ca91629cc331164e9f2"
server_key_version: "encryptionkey@0"
I20260929 02:54:03.805574 24417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:54:03.809079 24432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:03.810400 24417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:54:03.810540 24417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "86245bed97d74d5daff4e50f9a768a72"
format_stamp: "Formatted at 2026-09-29 02:54:03 on dist-test-slave-t6c1"
server_key: "554f7f5d3f5d93ca11a82a61f78c57b8"
server_key_iv: "9af7f74787d46ca91629cc331164e9f2"
server_key_version: "encryptionkey@0"
I20260929 02:54:03.810716 24417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:03.834781 24417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:03.838881 24417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:03.839174 24417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:03.840107 24417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:54:03.841382 24417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:54:03.841459 24417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:03.841544 24417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:54:03.841588 24417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:03.869592 24417 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:35525
I20260929 02:54:03.869652 24544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:35525 every 8 connection(s)
I20260929 02:54:03.871004 24417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:54:03.873018 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 24417
I20260929 02:54:03.873186 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:54:03.873626 4886 external_mini_cluster.cc:1428] Setting key 7f6555771577b9e03b82004bdda67d92
Sep 29 02:54:03 dist-test-slave-t6c1 krb5kdc[24177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650443, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.130@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:03.892879 24413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:03.873816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:47715 (local address 127.4.197.190:42847)
0929 02:54:03.874027 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:03.874034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:03.875332 (+ 1298us) server_negotiation.cc:407] Connection header received
0929 02:54:03.877825 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:03.877830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:03.877915 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:03.878001 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:03.879949 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:03.880645 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:03.882698 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:03.883285 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:03.883449 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:03.887330 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:03.887335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:03.887338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:03.887374 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:03.889749 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:03.890517 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:03.890523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:03.890526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:03.890684 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:03.891114 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:03.891118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:03.891121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:03.891453 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:03.891587 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:03.892328 (+ 741us) server_negotiation.cc:299] Negotiation successful
0929 02:54:03.892570 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:54:03.893783 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:03.874418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42847 (local address 127.4.197.130:47715)
0929 02:54:03.875190 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:03.875230 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:03.877523 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:03.878276 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:03.878288 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:03.879029 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:03.879793 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:03.879815 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:03.880780 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:03.880784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:03.882562 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:03.882575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:03.883747 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:03.883754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:03.884005 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:03.884632 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:03.884654 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:03.887109 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:03.889882 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:03.889891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:03.889908 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:03.890369 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:03.890784 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:03.890787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:03.890790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:03.891016 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:03.891568 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:03.891573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:03.891960 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 02:54:03.892186 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 02:54:03.892587 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":291,"threads_started":1}
I20260929 02:54:03.895149 24545 heartbeater.cc:347] Connected to a master server at 127.4.197.190:42847
I20260929 02:54:03.895433 24545 heartbeater.cc:464] Registering TS with master...
I20260929 02:54:03.896092 24545 heartbeater.cc:511] Master 127.4.197.190:42847 requested a full tablet report, sending...
I20260929 02:54:03.897454 24222 ts_manager.cc:194] Registered new tserver with Master: 86245bed97d74d5daff4e50f9a768a72 (127.4.197.130:35525)
I20260929 02:54:03.898257 24222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.130@KRBTEST.COM'} at 127.4.197.130:47715
WARNING: no policy specified for kudu/127.4.197.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
Entry for principal kudu/127.4.197.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab.
I20260929 02:54:03.924078 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42847
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:04.235960 24550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:04.236238 24550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:04.236356 24550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:04.243552 24550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:04.243680 24550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:04.243834 24550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:54:04.253504 24550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.131.keytab
--principal=kudu/127.4.197.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:42847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:04.254814 24550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:04.256155 24550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:04.268571 24555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:04.268528 24558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:04.268594 24556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:04.269594 24550 server_base.cc:1034] running on GCE node
I20260929 02:54:04.270223 24550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:04.271282 24550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:04.272488 24550 hybrid_clock.cc:648] HybridClock initialized: now 1790650444272445 us; error 37 us; skew 500 ppm
I20260929 02:54:04.272795 24550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:04 dist-test-slave-t6c1 krb5kdc[24177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650444, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:04.275840 24550 init.cc:377] Logged in from keytab as kudu/127.4.197.131@KRBTEST.COM (short username kudu)
I20260929 02:54:04.277400 24550 webserver.cc:460] Webserver started at http://127.4.197.131:46479/ using document root <none> and password file <none>
I20260929 02:54:04.278098 24550 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:04.278216 24550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:04.278512 24550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:04.281515 24550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "586d668c49ab49cb9ef53bf81255f683"
format_stamp: "Formatted at 2026-09-29 02:54:04 on dist-test-slave-t6c1"
server_key: "88931a5e72f5123b8783fc2ed93553c8"
server_key_iv: "abffe10e68069994671bbbf5fc0c7fa6"
server_key_version: "encryptionkey@0"
I20260929 02:54:04.282289 24550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "586d668c49ab49cb9ef53bf81255f683"
format_stamp: "Formatted at 2026-09-29 02:54:04 on dist-test-slave-t6c1"
server_key: "88931a5e72f5123b8783fc2ed93553c8"
server_key_iv: "abffe10e68069994671bbbf5fc0c7fa6"
server_key_version: "encryptionkey@0"
I20260929 02:54:04.287730 24550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:54:04.291116 24565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:04.292466 24550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:54:04.292627 24550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "586d668c49ab49cb9ef53bf81255f683"
format_stamp: "Formatted at 2026-09-29 02:54:04 on dist-test-slave-t6c1"
server_key: "88931a5e72f5123b8783fc2ed93553c8"
server_key_iv: "abffe10e68069994671bbbf5fc0c7fa6"
server_key_version: "encryptionkey@0"
I20260929 02:54:04.292806 24550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:04.307678 24550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:04.311200 24550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:04.311427 24550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:04.312256 24550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:54:04.313455 24550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:54:04.313525 24550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:04.313621 24550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:54:04.313663 24550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:04.343425 24550 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:38269
I20260929 02:54:04.343463 24677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:38269 every 8 connection(s)
I20260929 02:54:04.344689 24550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:54:04.346484 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 24550
I20260929 02:54:04.346640 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:54:04.347123 4886 external_mini_cluster.cc:1428] Setting key a2b9307458df3811ada9d604f31f79e2
Sep 29 02:54:04 dist-test-slave-t6c1 krb5kdc[24177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650444, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.131@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:04.363952 24413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.347243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:42063 (local address 127.4.197.190:42847)
0929 02:54:04.347467 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:04.347473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:04.348743 (+ 1270us) server_negotiation.cc:407] Connection header received
0929 02:54:04.350867 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:04.350871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:04.350973 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:04.351047 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:04.352514 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.353225 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.354712 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.355145 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.355234 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.358406 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:04.358411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:04.358414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:04.358447 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:04.361045 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:04.361806 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:04.361812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:04.361815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:04.361949 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:04.362370 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:04.362373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:04.362376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:04.362695 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:04.362805 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:04.363532 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 02:54:04.363705 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:54:04.364861 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.347773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:42847 (local address 127.4.197.131:42063)
0929 02:54:04.348607 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:04.348645 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:04.350607 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:04.351248 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:04.351256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:04.351790 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:04.352378 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.352396 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.353346 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.353349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.354602 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.354613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.355323 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.355328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.355523 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.356109 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:04.356133 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:04.358191 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:04.361175 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:04.361184 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:04.361202 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:04.361710 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:04.362062 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:04.362067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:04.362071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:04.362258 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:04.362809 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:04.362816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:04.363210 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 02:54:04.363447 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 02:54:04.363830 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":319,"threads_started":1}
I20260929 02:54:04.366182 24678 heartbeater.cc:347] Connected to a master server at 127.4.197.190:42847
I20260929 02:54:04.366446 24678 heartbeater.cc:464] Registering TS with master...
I20260929 02:54:04.367146 24678 heartbeater.cc:511] Master 127.4.197.190:42847 requested a full tablet report, sending...
I20260929 02:54:04.368567 24222 ts_manager.cc:194] Registered new tserver with Master: 586d668c49ab49cb9ef53bf81255f683 (127.4.197.131:38269)
I20260929 02:54:04.369400 24222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.197.131@KRBTEST.COM'} at 127.4.197.131:42063
I20260929 02:54:04.374610 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:54:04.391047 24413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.378674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.4.197.190:42847)
0929 02:54:04.378859 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:04.378866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:04.379286 (+ 420us) server_negotiation.cc:407] Connection header received
0929 02:54:04.379463 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:04.379467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:04.379530 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:04.379610 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:04.380964 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.381688 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.382952 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.383380 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.383545 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.385913 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:04.385918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:04.385922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:04.385954 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:04.388578 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:04.389179 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:04.389184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:04.389187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:04.389312 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:04.389649 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:04.389655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:04.389659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:04.389964 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:04.390185 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:04.390523 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 02:54:04.390729 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:54:04.394594 24222 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:54:04.401356 24222 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:54:04.415812 24411 heartbeater.cc:503] Master 127.4.197.190:42847 was elected leader, sending a full tablet report...
I20260929 02:54:04.423707 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.412917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:35525 (local address 127.0.0.1:53858)
0929 02:54:04.414962 (+ 2045us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:04.414979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:04.415090 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:04.416497 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:04.416504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:04.416533 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:04.416941 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.416954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.418964 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.418968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.421231 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.421249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.423012 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.423018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.423177 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.423207 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:54:04.423285 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:54:04.423402 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1893,"thread_start_us":2905,"threads_started":1}
I20260929 02:54:04.423753 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.413928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.4.197.129:45273)
0929 02:54:04.414557 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:04.414565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:04.416284 (+ 1719us) server_negotiation.cc:407] Connection header received
0929 02:54:04.416365 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:04.416371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:04.416597 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:04.416765 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:04.417666 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.419048 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.421190 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.422444 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.422573 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.422796 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:04.423249 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 02:54:04.423429 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":1599,"threads_started":1}
I20260929 02:54:04.424196 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.412730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:38269 (local address 127.0.0.1:42410)
0929 02:54:04.413582 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:04.413627 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:04.413801 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:04.417094 (+ 3293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:04.417098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:04.417126 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:04.417596 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.417608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.419582 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.419587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.421727 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.421740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.423482 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.423489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.423665 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.423699 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:54:04.423766 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 02:54:04.423885 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":1396,"threads_started":1}
I20260929 02:54:04.424528 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.413385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.4.197.130:35525)
0929 02:54:04.415687 (+ 2302us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:04.415703 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:04.415734 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:54:04.415845 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:04.415852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:04.416120 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:04.416312 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:04.417271 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.418801 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.421419 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.422790 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.422955 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.423140 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:04.424135 (+ 995us) server_negotiation.cc:299] Negotiation successful
0929 02:54:04.424266 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1977,"thread_start_us":1895,"threads_started":1}
I20260929 02:54:04.423669 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.413274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:45273 (local address 127.0.0.1:41482)
0929 02:54:04.416056 (+ 2782us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:04.416072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:04.416181 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:04.416986 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:04.416990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:04.417010 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:04.417498 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.417512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.419185 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.419191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.421015 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.421027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.422620 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.422626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.422804 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.422855 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 02:54:04.423018 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 02:54:04.423203 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2664,"mutex_wait_us":39,"thread_start_us":1478,"threads_started":1}
I20260929 02:54:04.424734 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.412952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.4.197.131:38269)
0929 02:54:04.416549 (+ 3597us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:04.416559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:04.416592 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:54:04.416678 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:04.416685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:04.416895 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:04.417797 (+ 902us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:04.418042 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.419453 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.421907 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.423271 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.424041 (+ 770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.424251 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:04.424319 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 02:54:04.424492 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3287,"thread_start_us":4441,"threads_started":1}
I20260929 02:54:04.428066 24346 tablet_service.cc:1467] Processing CreateTablet for tablet 3979705165f94a52a4ccc91c95bb40ba (DEFAULT_TABLE table=test-table [id=84efa20c9d23473d983fbae777bc5f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:54:04.428093 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 3979705165f94a52a4ccc91c95bb40ba (DEFAULT_TABLE table=test-table [id=84efa20c9d23473d983fbae777bc5f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:54:04.428094 24480 tablet_service.cc:1467] Processing CreateTablet for tablet 3979705165f94a52a4ccc91c95bb40ba (DEFAULT_TABLE table=test-table [id=84efa20c9d23473d983fbae777bc5f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:54:04.429379 24346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3979705165f94a52a4ccc91c95bb40ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:04.429411 24480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3979705165f94a52a4ccc91c95bb40ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:04.429425 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3979705165f94a52a4ccc91c95bb40ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:04.440429 24696 tablet_bootstrap.cc:492] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312: Bootstrap starting.
I20260929 02:54:04.443132 24697 tablet_bootstrap.cc:492] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: Bootstrap starting.
I20260929 02:54:04.443825 24696 tablet_bootstrap.cc:654] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:04.445300 24696 log.cc:824] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:04.446857 24697 tablet_bootstrap.cc:654] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:04.447343 24698 tablet_bootstrap.cc:492] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: Bootstrap starting.
I20260929 02:54:04.448082 24697 log.cc:824] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:04.448452 24696 tablet_bootstrap.cc:492] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312: No bootstrap required, opened a new log
I20260929 02:54:04.448747 24696 ts_tablet_manager.cc:1397] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:54:04.449985 24698 tablet_bootstrap.cc:654] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:04.451030 24698 log.cc:824] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:04.454280 24698 tablet_bootstrap.cc:492] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: No bootstrap required, opened a new log
I20260929 02:54:04.454286 24697 tablet_bootstrap.cc:492] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: No bootstrap required, opened a new log
I20260929 02:54:04.454577 24697 ts_tablet_manager.cc:1397] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260929 02:54:04.454816 24698 ts_tablet_manager.cc:1397] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:54:04.459234 24696 raft_consensus.cc:348] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586d668c49ab49cb9ef53bf81255f683" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38269 } } peers { permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 } } peers { permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 } }
I20260929 02:54:04.459684 24696 raft_consensus.cc:374] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:04.459820 24696 raft_consensus.cc:729] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25763214aa94c96a1a13cd9e3a63312, State: Initialized, Role: FOLLOWER
I20260929 02:54:04.460484 24696 consensus_queue.cc:260] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586d668c49ab49cb9ef53bf81255f683" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38269 } } peers { permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 } } peers { permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 } }
I20260929 02:54:04.462754 24697 raft_consensus.cc:348] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586d668c49ab49cb9ef53bf81255f683" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38269 } } peers { permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 } } peers { permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 } }
I20260929 02:54:04.463316 24697 raft_consensus.cc:374] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:04.463435 24697 raft_consensus.cc:729] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86245bed97d74d5daff4e50f9a768a72, State: Initialized, Role: FOLLOWER
I20260929 02:54:04.464113 24697 consensus_queue.cc:260] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586d668c49ab49cb9ef53bf81255f683" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38269 } } peers { permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 } } peers { permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 } }
I20260929 02:54:04.465616 24698 raft_consensus.cc:348] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586d668c49ab49cb9ef53bf81255f683" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38269 } } peers { permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 } } peers { permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 } }
I20260929 02:54:04.466133 24698 raft_consensus.cc:374] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:04.466238 24698 raft_consensus.cc:729] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 586d668c49ab49cb9ef53bf81255f683, State: Initialized, Role: FOLLOWER
I20260929 02:54:04.466976 24698 consensus_queue.cc:260] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586d668c49ab49cb9ef53bf81255f683" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38269 } } peers { permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 } } peers { permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 } }
I20260929 02:54:04.468149 24678 heartbeater.cc:503] Master 127.4.197.190:42847 was elected leader, sending a full tablet report...
I20260929 02:54:04.468741 24696 ts_tablet_manager.cc:1428] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260929 02:54:04.468811 24698 ts_tablet_manager.cc:1428] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 02:54:04.469762 24545 heartbeater.cc:503] Master 127.4.197.190:42847 was elected leader, sending a full tablet report...
I20260929 02:54:04.470463 24697 ts_tablet_manager.cc:1428] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
W20260929 02:54:04.596246 24679 tablet.cc:2369] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:54:04.623152 24546 tablet.cc:2369] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:54:04.628644 24704 raft_consensus.cc:482] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:54:04.628944 24704 raft_consensus.cc:504] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586d668c49ab49cb9ef53bf81255f683" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38269 } } peers { permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 } } peers { permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 } }
I20260929 02:54:04.630607 24704 leader_election.cc:288] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86245bed97d74d5daff4e50f9a768a72 (127.4.197.130:35525), e25763214aa94c96a1a13cd9e3a63312 (127.4.197.129:45273)
I20260929 02:54:04.637187 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.631139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:57131 (local address 127.4.197.129:45273)
0929 02:54:04.631364 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:04.631370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:04.631391 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:54:04.631481 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:04.631484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:04.631542 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:04.631615 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:04.632039 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.633032 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.635372 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.636470 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.636618 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.636692 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:04.636883 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 02:54:04.637012 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:54:04.637274 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.630863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:45273 (local address 127.4.197.131:57131)
0929 02:54:04.631104 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:04.631130 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:04.631267 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:04.631668 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:04.631673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:04.631704 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:04.631916 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.631925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.633153 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.633156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.635241 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.635255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.636615 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.636619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.636747 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.636770 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:54:04.636844 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:54:04.636989 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":80}
I20260929 02:54:04.637600 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.631338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:47435 (local address 127.4.197.130:35525)
0929 02:54:04.631534 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:04.631540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:04.631724 (+ 184us) server_negotiation.cc:407] Connection header received
0929 02:54:04.631859 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:04.631864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:04.631948 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:04.632031 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:04.632465 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.633351 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.635638 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.636768 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.636892 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.636965 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:04.637261 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 02:54:04.637379 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:54:04.637701 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.630959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:35525 (local address 127.4.197.131:47435)
0929 02:54:04.631611 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:04.631627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:04.631732 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:04.632054 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:04.632057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:04.632075 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:04.632325 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.632332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.633475 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.633480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.635503 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:04.635514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.636951 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.636957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:04.637107 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.637141 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:54:04.637223 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 02:54:04.637338 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":55,"thread_start_us":738,"threads_started":1}
W20260929 02:54:04.638032 24412 tablet.cc:2369] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:54:04.638392 24366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3979705165f94a52a4ccc91c95bb40ba" candidate_uuid: "586d668c49ab49cb9ef53bf81255f683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25763214aa94c96a1a13cd9e3a63312" is_pre_election: true
I20260929 02:54:04.638688 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3979705165f94a52a4ccc91c95bb40ba" candidate_uuid: "586d668c49ab49cb9ef53bf81255f683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86245bed97d74d5daff4e50f9a768a72" is_pre_election: true
I20260929 02:54:04.638990 24366 raft_consensus.cc:2454] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 586d668c49ab49cb9ef53bf81255f683 in term 0.
I20260929 02:54:04.639181 24500 raft_consensus.cc:2454] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 586d668c49ab49cb9ef53bf81255f683 in term 0.
I20260929 02:54:04.639598 24568 leader_election.cc:302] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 586d668c49ab49cb9ef53bf81255f683, e25763214aa94c96a1a13cd9e3a63312; no voters:
I20260929 02:54:04.639918 24704 raft_consensus.cc:2781] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:54:04.640038 24704 raft_consensus.cc:482] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:54:04.640134 24704 raft_consensus.cc:3037] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:04.642189 24704 raft_consensus.cc:504] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586d668c49ab49cb9ef53bf81255f683" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38269 } } peers { permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 } } peers { permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 } }
I20260929 02:54:04.642760 24704 leader_election.cc:288] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [CANDIDATE]: Term 1 election: Requested vote from peers 86245bed97d74d5daff4e50f9a768a72 (127.4.197.130:35525), e25763214aa94c96a1a13cd9e3a63312 (127.4.197.129:45273)
I20260929 02:54:04.643160 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3979705165f94a52a4ccc91c95bb40ba" candidate_uuid: "586d668c49ab49cb9ef53bf81255f683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86245bed97d74d5daff4e50f9a768a72"
I20260929 02:54:04.643231 24366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3979705165f94a52a4ccc91c95bb40ba" candidate_uuid: "586d668c49ab49cb9ef53bf81255f683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25763214aa94c96a1a13cd9e3a63312"
I20260929 02:54:04.643313 24500 raft_consensus.cc:3037] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:04.643430 24366 raft_consensus.cc:3037] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:04.645488 24366 raft_consensus.cc:2454] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 586d668c49ab49cb9ef53bf81255f683 in term 1.
I20260929 02:54:04.645537 24500 raft_consensus.cc:2454] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 586d668c49ab49cb9ef53bf81255f683 in term 1.
I20260929 02:54:04.645845 24566 leader_election.cc:302] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 586d668c49ab49cb9ef53bf81255f683, 86245bed97d74d5daff4e50f9a768a72; no voters:
I20260929 02:54:04.646134 24704 raft_consensus.cc:2781] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:04.646514 24704 raft_consensus.cc:686] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 1 LEADER]: Becoming Leader. State: Replica: 586d668c49ab49cb9ef53bf81255f683, State: Running, Role: LEADER
I20260929 02:54:04.646958 24704 consensus_queue.cc:237] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586d668c49ab49cb9ef53bf81255f683" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38269 } } peers { permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 } } peers { permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 } }
I20260929 02:54:04.651185 24221 catalog_manager.cc:5495] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 reported cstate change: term changed from 0 to 1, leader changed from <none> to 586d668c49ab49cb9ef53bf81255f683 (127.4.197.131). New cstate: current_term: 1 leader_uuid: "586d668c49ab49cb9ef53bf81255f683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586d668c49ab49cb9ef53bf81255f683" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 38269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 } health_report { overall_health: UNKNOWN } } }
I20260929 02:54:04.720650 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.715291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.4.197.131:38269)
0929 02:54:04.715489 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:04.715495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:04.715515 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:54:04.715681 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:04.715685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:04.715777 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:04.715910 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:04.716278 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.717012 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.718423 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.718841 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.718997 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.719478 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:04.720057 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:04.720191 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:04.720293 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 02:54:04.720390 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:54:04.729218 24500 raft_consensus.cc:1261] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72 [term 1 FOLLOWER]: Refusing update from remote peer 586d668c49ab49cb9ef53bf81255f683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:54:04.729218 24366 raft_consensus.cc:1261] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312 [term 1 FOLLOWER]: Refusing update from remote peer 586d668c49ab49cb9ef53bf81255f683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:54:04.730242 24706 consensus_queue.cc:1035] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e25763214aa94c96a1a13cd9e3a63312" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 45273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:54:04.730584 24704 consensus_queue.cc:1035] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86245bed97d74d5daff4e50f9a768a72" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:54:04.763986 24221 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:54:04.764266 24221 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:54:04.768067 24221 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a95fe50323454366973faf61ac8434a1: Sending DeleteTablet for 3 replicas of tablet 3979705165f94a52a4ccc91c95bb40ba
I20260929 02:54:04.769291 24613 tablet_service.cc:1514] Processing DeleteTablet for tablet 3979705165f94a52a4ccc91c95bb40ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:54:04 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:42410
I20260929 02:54:04.769337 24480 tablet_service.cc:1514] Processing DeleteTablet for tablet 3979705165f94a52a4ccc91c95bb40ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:54:04 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:53858
I20260929 02:54:04.769625 24346 tablet_service.cc:1514] Processing DeleteTablet for tablet 3979705165f94a52a4ccc91c95bb40ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:54:04 UTC) from {username='kudu', principal='kudu/127.4.197.190@KRBTEST.COM'} at 127.0.0.1:41482
I20260929 02:54:04.771258 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 24283
I20260929 02:54:04.781260 24719 tablet_replica.cc:330] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: stopping tablet replica
I20260929 02:54:04.781991 24719 raft_consensus.cc:2229] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:54:04.782697 24719 raft_consensus.cc:2258] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:54:04.783422 24720 tablet_replica.cc:330] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: stopping tablet replica
I20260929 02:54:04.784325 24720 raft_consensus.cc:2229] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:54:04.786697 24719 ts_tablet_manager.cc:1905] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:54:04.787541 24720 raft_consensus.cc:2258] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:54:04.790146 24720 ts_tablet_manager.cc:1905] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:54:04 dist-test-slave-t6c1 krb5kdc[24177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.129@KRBTEST.COM
I20260929 02:54:04.794003 24721 tablet_replica.cc:330] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312: stopping tablet replica
I20260929 02:54:04.794550 24721 raft_consensus.cc:2229] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:54:04.795233 24721 raft_consensus.cc:2258] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:54:04.797628 24721 ts_tablet_manager.cc:1905] T 3979705165f94a52a4ccc91c95bb40ba P e25763214aa94c96a1a13cd9e3a63312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:54:04.801596 24720 ts_tablet_manager.cc:1918] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:54:04.801775 24720 log.cc:1196] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/3979705165f94a52a4ccc91c95bb40ba
I20260929 02:54:04.802289 24720 ts_tablet_manager.cc:1939] T 3979705165f94a52a4ccc91c95bb40ba P 586d668c49ab49cb9ef53bf81255f683: Deleting consensus metadata
I20260929 02:54:04.802867 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.774097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.4.197.129:45273)
0929 02:54:04.774282 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:04.774288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:04.778244 (+ 3956us) server_negotiation.cc:407] Connection header received
0929 02:54:04.778451 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:04.778457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:04.778558 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:04.778684 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:04.784931 (+ 6247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.786158 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.788088 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.788776 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.788883 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.794684 (+ 5801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:04.794701 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:04.794719 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:04.794770 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:04.799306 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:04.800220 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:04.800229 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:04.800232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:04.800433 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:04.800932 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:04.800937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:04.800940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:04.801406 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:04.801681 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:04.802251 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 02:54:04.802497 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:54:04.804152 24209 catalog_manager.cc:4841] TS 586d668c49ab49cb9ef53bf81255f683 (127.4.197.131:38269): tablet 3979705165f94a52a4ccc91c95bb40ba (table test-table [id=84efa20c9d23473d983fbae777bc5f1f]) successfully deleted
I20260929 02:54:04.804281 24406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:04.805603 24719 ts_tablet_manager.cc:1918] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:54:04.805768 24719 log.cc:1196] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/3979705165f94a52a4ccc91c95bb40ba
I20260929 02:54:04.806310 24719 ts_tablet_manager.cc:1939] T 3979705165f94a52a4ccc91c95bb40ba P 86245bed97d74d5daff4e50f9a768a72: Deleting consensus metadata
I20260929 02:54:04.807961 24206 catalog_manager.cc:4841] TS 86245bed97d74d5daff4e50f9a768a72 (127.4.197.130:35525): tablet 3979705165f94a52a4ccc91c95bb40ba (table test-table [id=84efa20c9d23473d983fbae777bc5f1f]) successfully deleted
I20260929 02:54:04.925995 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 24283
W20260929 02:54:04.946722 24208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.197.129:45273 (error 108)
I20260929 02:54:04.947400 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 24417
I20260929 02:54:04.949285 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.948461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:45273 (local address 127.0.0.1:41502)
0929 02:54:04.948944 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:04.949078 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":222,"threads_started":1}
W20260929 02:54:04.949577 24208 catalog_manager.cc:4584] TS e25763214aa94c96a1a13cd9e3a63312 (127.4.197.129:45273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3979705165f94a52a4ccc91c95bb40ba: Network error: Client connection negotiation failed: client connection to 127.4.197.129:45273: connect: Connection refused (error 111)
Sep 29 02:54:04 dist-test-slave-t6c1 krb5kdc[24177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.130@KRBTEST.COM
I20260929 02:54:04.964437 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:04.948176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.4.197.130:35525)
0929 02:54:04.948424 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:04.948431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:04.948451 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:54:04.948622 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:04.948628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:04.948718 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:04.948822 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:04.950525 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.951331 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.952595 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:04.952994 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:04.953105 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:04.957635 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:04.957656 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:04.957676 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:04.957729 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:04.961379 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:04.962221 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:04.962232 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:04.962236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:04.962396 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:04.962744 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:04.962748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:04.962752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:04.963295 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:04.963597 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:04.963802 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 02:54:04.964027 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 02:54:04.965082 24540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:05.003446 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:05.002887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:45273 (local address 127.0.0.1:41516)
0929 02:54:05.003150 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:05.003260 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 02:54:05.004626 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:05.004201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:45273 (local address 127.0.0.1:41530)
0929 02:54:05.004357 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:05.004460 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 02:54:05.057497 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 24417
I20260929 02:54:05.065212 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:05.064618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:45273 (local address 127.0.0.1:41534)
0929 02:54:05.064886 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:05.065020 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 02:54:05.066483 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:05.066023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:45273 (local address 127.0.0.1:41550)
0929 02:54:05.066203 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:05.066299 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 02:54:05.072782 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 24550
Sep 29 02:54:05 dist-test-slave-t6c1 krb5kdc[24177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.131@KRBTEST.COM
I20260929 02:54:05.091240 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:05.074026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.4.197.131:38269)
0929 02:54:05.074177 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:05.074181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:05.074196 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:54:05.074250 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:05.074254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:05.074334 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:05.074414 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:05.076158 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:05.077057 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:05.078879 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:05.079421 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:05.079530 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:05.084955 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:05.084967 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:05.084982 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:05.085029 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:05.088339 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:05.089162 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:05.089169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:05.089171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:05.089351 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:05.089760 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:05.089763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:05.089765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:05.090117 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:05.090396 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:05.090714 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 02:54:05.090902 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 02:54:05.092090 24673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:05.147501 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:05.146993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:45273 (local address 127.0.0.1:41552)
0929 02:54:05.147213 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:05.147316 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 02:54:05.148746 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:05.148271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:45273 (local address 127.0.0.1:41558)
0929 02:54:05.148515 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:05.148598 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260929 02:54:05.216075 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 24550
I20260929 02:54:05.231441 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 24190
I20260929 02:54:05.232661 24252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:05.342043 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:05.341374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:45273 (local address 127.0.0.1:41560)
0929 02:54:05.341731 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:05.341837 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 02:54:05.342399 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 24190
2026-09-29T02:54:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4308 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T02:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:05Z Disabled control of system clock
I20260929 02:54:05.409608 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33505
--webserver_interface=127.4.197.190
--webserver_port=0
--builtin_ntp_servers=127.4.197.148:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:33505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:54:05.730244 24732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:05.730544 24732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:05.730676 24732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:05.738788 24732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:05.738904 24732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:05.738996 24732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:05.739073 24732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:05.739130 24732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:05.748373 24732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:33505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:05.749668 24732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:05.751444 24732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:05.762317 24737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:05.762387 24740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:05.762399 24738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:05.763136 24732 server_base.cc:1034] running on GCE node
I20260929 02:54:05.763818 24732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:05.765139 24732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:05.766335 24732 hybrid_clock.cc:648] HybridClock initialized: now 1790650445766295 us; error 28 us; skew 500 ppm
I20260929 02:54:05.766654 24732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:54:05.768853 24732 webserver.cc:460] Webserver started at http://127.4.197.190:39357/ using document root <none> and password file <none>
I20260929 02:54:05.769557 24732 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:05.769660 24732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:05.769979 24732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:05.773291 24732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7"
format_stamp: "Formatted at 2026-09-29 02:54:05 on dist-test-slave-t6c1"
server_key: "392a1145928e2dedcb2ac2909926f555"
server_key_iv: "903881a14ea0c541e6699e2eedd6fe5d"
server_key_version: "encryptionkey@0"
I20260929 02:54:05.774096 24732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7"
format_stamp: "Formatted at 2026-09-29 02:54:05 on dist-test-slave-t6c1"
server_key: "392a1145928e2dedcb2ac2909926f555"
server_key_iv: "903881a14ea0c541e6699e2eedd6fe5d"
server_key_version: "encryptionkey@0"
I20260929 02:54:05.779230 24732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:54:05.782537 24746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:05.784015 24732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:54:05.784191 24732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7"
format_stamp: "Formatted at 2026-09-29 02:54:05 on dist-test-slave-t6c1"
server_key: "392a1145928e2dedcb2ac2909926f555"
server_key_iv: "903881a14ea0c541e6699e2eedd6fe5d"
server_key_version: "encryptionkey@0"
I20260929 02:54:05.784396 24732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:05.803218 24732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:05.804136 24732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:05.804425 24732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:05.821482 24732 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:33505
I20260929 02:54:05.821502 24797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:33505 every 8 connection(s)
I20260929 02:54:05.822893 24732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:05.827111 24798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:05.828882 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 24732
I20260929 02:54:05.829056 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:05.829528 4886 external_mini_cluster.cc:1428] Setting key 13003b6fb8a407c7e100e8bab30cdf7f
I20260929 02:54:05.841361 24798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7: Bootstrap starting.
I20260929 02:54:05.845388 24798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:05.846077 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:05.831669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.4.197.190:33505)
0929 02:54:05.832867 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:05.832882 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:05.832942 (+ 60us) server_negotiation.cc:407] Connection header received
0929 02:54:05.835420 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:05.835450 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:05.836099 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:05.836597 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:05.837198 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:05.839269 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:05.840672 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:05.841767 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:05.841962 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:05.843156 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:05.843165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:05.843190 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:05.843928 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:05.844043 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:05.844070 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:05.844186 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:05.844411 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 02:54:05.844759 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":361,"threads_started":1}
I20260929 02:54:05.846638 24798 log.cc:824] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:05.849346 24798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7: No bootstrap required, opened a new log
I20260929 02:54:05.856398 24798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33505 } }
I20260929 02:54:05.856755 24798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:05.856846 24798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fb2ab0d90b49fe834309c1e5b4f7d7, State: Initialized, Role: FOLLOWER
I20260929 02:54:05.857419 24798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33505 } }
I20260929 02:54:05.857637 24798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:05.857760 24798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:05.857918 24798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:05.860054 24798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33505 } }
I20260929 02:54:05.860507 24798 leader_election.cc:302] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07fb2ab0d90b49fe834309c1e5b4f7d7; no voters:
I20260929 02:54:05.861033 24798 leader_election.cc:288] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:05.861207 24803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:05.862214 24803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [term 1 LEADER]: Becoming Leader. State: Replica: 07fb2ab0d90b49fe834309c1e5b4f7d7, State: Running, Role: LEADER
I20260929 02:54:05.862807 24803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33505 } }
I20260929 02:54:05.863243 24798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:05.865509 24805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07fb2ab0d90b49fe834309c1e5b4f7d7. Latest consensus state: current_term: 1 leader_uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33505 } } }
I20260929 02:54:05.865486 24804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fb2ab0d90b49fe834309c1e5b4f7d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33505 } } }
I20260929 02:54:05.865948 24805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:05.865947 24804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:05.867621 24810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:05.873034 24810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:05.884214 24810 catalog_manager.cc:1347] Generated new cluster ID: ee5fd4175e314789b7b2170807e50443
I20260929 02:54:05.884369 24810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:05.906781 24810 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:05.908087 24810 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:05.919369 24810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7: Generated new TSK 0
I20260929 02:54:05.920176 24810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:05.927785 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.129:0
--local_ip_for_outbound_sockets=127.4.197.129
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33505
--builtin_ntp_servers=127.4.197.148:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:54:06.291249 24822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:06.291532 24822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:06.291626 24822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:06.298808 24822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:06.298905 24822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:06.299050 24822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.129
I20260929 02:54:06.307919 24822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.197.129
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.129
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:06.309037 24822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:06.310652 24822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:06.322345 24827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:06.322381 24830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:06.322424 24828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:06.323459 24822 server_base.cc:1034] running on GCE node
I20260929 02:54:06.324012 24822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:06.324944 24822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:06.326153 24822 hybrid_clock.cc:648] HybridClock initialized: now 1790650446326112 us; error 27 us; skew 500 ppm
I20260929 02:54:06.326470 24822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:54:06.328783 24822 webserver.cc:460] Webserver started at http://127.4.197.129:37397/ using document root <none> and password file <none>
I20260929 02:54:06.329476 24822 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:06.329578 24822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:06.329893 24822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:06.332877 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/data/instance:
uuid: "2d35aa4c54e04f438c99eabd0c393fa0"
format_stamp: "Formatted at 2026-09-29 02:54:06 on dist-test-slave-t6c1"
server_key: "5cbc5efdd398b8e83b7d35057180f021"
server_key_iv: "6b9cf5b45dc84ad1f45705e72391e164"
server_key_version: "encryptionkey@0"
I20260929 02:54:06.333678 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance:
uuid: "2d35aa4c54e04f438c99eabd0c393fa0"
format_stamp: "Formatted at 2026-09-29 02:54:06 on dist-test-slave-t6c1"
server_key: "5cbc5efdd398b8e83b7d35057180f021"
server_key_iv: "6b9cf5b45dc84ad1f45705e72391e164"
server_key_version: "encryptionkey@0"
I20260929 02:54:06.338964 24822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:54:06.342633 24836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:06.344045 24822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:54:06.344192 24822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/wal
uuid: "2d35aa4c54e04f438c99eabd0c393fa0"
format_stamp: "Formatted at 2026-09-29 02:54:06 on dist-test-slave-t6c1"
server_key: "5cbc5efdd398b8e83b7d35057180f021"
server_key_iv: "6b9cf5b45dc84ad1f45705e72391e164"
server_key_version: "encryptionkey@0"
I20260929 02:54:06.344405 24822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:06.366168 24822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:06.367070 24822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:06.367355 24822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:06.368286 24822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:54:06.369748 24822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:54:06.369841 24822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:06.369944 24822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:54:06.370013 24822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:06.399762 24822 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.129:42101
I20260929 02:54:06.399818 24948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.129:42101 every 8 connection(s)
I20260929 02:54:06.401010 24822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/data/info.pb
I20260929 02:54:06.405205 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 24822
I20260929 02:54:06.405381 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-0/wal/instance
I20260929 02:54:06.405820 4886 external_mini_cluster.cc:1428] Setting key 769674d7f9b292c211571f2f5baada0b
I20260929 02:54:06.409036 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.130:0
--local_ip_for_outbound_sockets=127.4.197.130
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33505
--builtin_ntp_servers=127.4.197.148:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:54:06.414727 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:06.403491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.129:41715 (local address 127.4.197.190:33505)
0929 02:54:06.403901 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:06.403906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:06.405218 (+ 1312us) server_negotiation.cc:407] Connection header received
0929 02:54:06.407275 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:06.407279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:06.407350 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:06.407448 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:06.408855 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:06.409651 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:06.411910 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:06.412325 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:06.412420 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:06.414007 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:06.414012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:06.414015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:06.414036 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:06.414081 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:06.414093 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:06.414161 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:06.414373 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 02:54:06.414463 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":186,"threads_started":1}
I20260929 02:54:06.416262 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:06.404095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33505 (local address 127.4.197.129:41715)
0929 02:54:06.405046 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:06.405092 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:06.406965 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:06.407709 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:06.407719 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:06.407820 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:06.408670 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:06.408698 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:06.409821 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:06.409835 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:06.411770 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:06.411784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:06.412534 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:06.412541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:06.412851 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:06.413629 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:54:06.413632 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:06.413690 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:54:06.413808 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:06.414171 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:06.414180 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:06.414187 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:54:06.414370 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 02:54:06.414795 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":238,"threads_started":1}
I20260929 02:54:06.417848 24949 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33505
I20260929 02:54:06.418196 24949 heartbeater.cc:464] Registering TS with master...
I20260929 02:54:06.419051 24949 heartbeater.cc:511] Master 127.4.197.190:33505 requested a full tablet report, sending...
I20260929 02:54:06.421144 24763 ts_manager.cc:194] Registered new tserver with Master: 2d35aa4c54e04f438c99eabd0c393fa0 (127.4.197.129:42101)
I20260929 02:54:06.422667 24763 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.129:41715
W20260929 02:54:06.728246 24953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:06.728542 24953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:06.728663 24953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:06.735908 24953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:06.736017 24953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:06.736167 24953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.130
I20260929 02:54:06.745338 24953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.197.130
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.130
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:06.746654 24953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:06.748502 24953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:06.761356 24958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:06.761391 24961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:06.761322 24959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:06.762466 24953 server_base.cc:1034] running on GCE node
I20260929 02:54:06.763128 24953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:06.764147 24953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:06.765336 24953 hybrid_clock.cc:648] HybridClock initialized: now 1790650446765288 us; error 39 us; skew 500 ppm
I20260929 02:54:06.765632 24953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:54:06.767841 24953 webserver.cc:460] Webserver started at http://127.4.197.130:38491/ using document root <none> and password file <none>
I20260929 02:54:06.768491 24953 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:06.768579 24953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:06.768877 24953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:06.771729 24953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/data/instance:
uuid: "4d7ef05d84834bfbbab9f90628f03b25"
format_stamp: "Formatted at 2026-09-29 02:54:06 on dist-test-slave-t6c1"
server_key: "6f2b6338bf9d1e8b4525c0274ffbba0c"
server_key_iv: "4b7a8e03751428802833266093c94eb0"
server_key_version: "encryptionkey@0"
I20260929 02:54:06.772495 24953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance:
uuid: "4d7ef05d84834bfbbab9f90628f03b25"
format_stamp: "Formatted at 2026-09-29 02:54:06 on dist-test-slave-t6c1"
server_key: "6f2b6338bf9d1e8b4525c0274ffbba0c"
server_key_iv: "4b7a8e03751428802833266093c94eb0"
server_key_version: "encryptionkey@0"
I20260929 02:54:06.777796 24953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:54:06.781365 24967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:06.782799 24953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 02:54:06.783046 24953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/wal
uuid: "4d7ef05d84834bfbbab9f90628f03b25"
format_stamp: "Formatted at 2026-09-29 02:54:06 on dist-test-slave-t6c1"
server_key: "6f2b6338bf9d1e8b4525c0274ffbba0c"
server_key_iv: "4b7a8e03751428802833266093c94eb0"
server_key_version: "encryptionkey@0"
I20260929 02:54:06.783313 24953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:06.809144 24953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:06.809885 24953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:06.810135 24953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:06.811026 24953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:54:06.812336 24953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:54:06.812417 24953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:06.812497 24953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:54:06.812549 24953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:06.845423 24953 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.130:43783
I20260929 02:54:06.845481 25079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.130:43783 every 8 connection(s)
I20260929 02:54:06.846662 24953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/data/info.pb
I20260929 02:54:06.855074 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 24953
I20260929 02:54:06.855254 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/wal/instance
I20260929 02:54:06.855808 4886 external_mini_cluster.cc:1428] Setting key 4501491295b734a16f0fea0d65d19026
I20260929 02:54:06.858707 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.197.131:0
--local_ip_for_outbound_sockets=127.4.197.131
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33505
--builtin_ntp_servers=127.4.197.148:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:54:06.859659 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:06.849276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:54089 (local address 127.4.197.190:33505)
0929 02:54:06.849483 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:06.849489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:06.850865 (+ 1376us) server_negotiation.cc:407] Connection header received
0929 02:54:06.852634 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:06.852639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:06.852708 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:06.852803 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:06.853842 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:06.854557 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:06.856808 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:06.857258 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:06.857374 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:06.858844 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:06.858851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:06.858855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:06.858876 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:06.858913 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:06.858968 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:06.859048 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:06.859254 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 02:54:06.859358 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:54:06.860756 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:06.849838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33505 (local address 127.4.197.130:54089)
0929 02:54:06.850719 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:06.850754 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:06.852384 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:06.853008 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:06.853016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:06.853105 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:06.853665 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:06.853682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:06.854734 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:06.854739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:06.856649 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:06.856666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:06.857512 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:06.857520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:06.857817 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:06.858519 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:54:06.858521 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:06.858570 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:54:06.858656 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:06.859049 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:06.859061 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:06.859069 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:54:06.859241 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 02:54:06.859675 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":254,"threads_started":1}
I20260929 02:54:06.862128 25080 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33505
I20260929 02:54:06.862444 25080 heartbeater.cc:464] Registering TS with master...
I20260929 02:54:06.863336 25080 heartbeater.cc:511] Master 127.4.197.190:33505 requested a full tablet report, sending...
I20260929 02:54:06.864835 24763 ts_manager.cc:194] Registered new tserver with Master: 4d7ef05d84834bfbbab9f90628f03b25 (127.4.197.130:43783)
I20260929 02:54:06.865826 24763 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.130:54089
W20260929 02:54:07.172333 25083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:07.172609 25083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:07.172721 25083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:07.179972 25083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:07.180076 25083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:07.180225 25083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.197.131
I20260929 02:54:07.189286 25083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.197.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.197.131
--webserver_port=0
--tserver_master_addrs=127.4.197.190:33505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.197.131
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:07.190474 25083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:07.192315 25083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:07.205240 25088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:07.205307 25089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:07.205284 25091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:07.206048 25083 server_base.cc:1034] running on GCE node
I20260929 02:54:07.206619 25083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:07.207564 25083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:07.208760 25083 hybrid_clock.cc:648] HybridClock initialized: now 1790650447208710 us; error 46 us; skew 500 ppm
I20260929 02:54:07.209062 25083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:54:07.211205 25083 webserver.cc:460] Webserver started at http://127.4.197.131:45197/ using document root <none> and password file <none>
I20260929 02:54:07.211835 25083 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:07.211943 25083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:07.212234 25083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:07.214982 25083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/data/instance:
uuid: "5daa3118f34d4b3d938f808c5757ec6b"
format_stamp: "Formatted at 2026-09-29 02:54:07 on dist-test-slave-t6c1"
server_key: "ce076bdec32302190006a4773580be82"
server_key_iv: "bef9086053dc47c0d3ffe5178e086860"
server_key_version: "encryptionkey@0"
I20260929 02:54:07.215725 25083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance:
uuid: "5daa3118f34d4b3d938f808c5757ec6b"
format_stamp: "Formatted at 2026-09-29 02:54:07 on dist-test-slave-t6c1"
server_key: "ce076bdec32302190006a4773580be82"
server_key_iv: "bef9086053dc47c0d3ffe5178e086860"
server_key_version: "encryptionkey@0"
I20260929 02:54:07.220443 25083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:54:07.223632 25097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:07.224822 25083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:54:07.224960 25083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/wal
uuid: "5daa3118f34d4b3d938f808c5757ec6b"
format_stamp: "Formatted at 2026-09-29 02:54:07 on dist-test-slave-t6c1"
server_key: "ce076bdec32302190006a4773580be82"
server_key_iv: "bef9086053dc47c0d3ffe5178e086860"
server_key_version: "encryptionkey@0"
I20260929 02:54:07.225138 25083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:07.235953 25083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:07.236649 25083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:07.236903 25083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:07.237792 25083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:54:07.239192 25083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:54:07.239286 25083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:07.239392 25083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:54:07.239459 25083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:07.269722 25083 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.131:40721
I20260929 02:54:07.269791 25209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.131:40721 every 8 connection(s)
I20260929 02:54:07.271052 25083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/data/info.pb
I20260929 02:54:07.272238 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 25083
I20260929 02:54:07.272401 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/wal/instance
I20260929 02:54:07.272977 4886 external_mini_cluster.cc:1428] Setting key e42d41f4e90928332a2c8e5d1faa94a8
I20260929 02:54:07.284606 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.273821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.131:34399 (local address 127.4.197.190:33505)
0929 02:54:07.274028 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.274034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.275764 (+ 1730us) server_negotiation.cc:407] Connection header received
0929 02:54:07.277742 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.277748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.277842 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.277944 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:07.279191 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.279960 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.281720 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.282159 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.282291 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.283708 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:07.283715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:07.283720 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:07.283743 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:07.283777 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:07.283789 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:07.283874 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.284107 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.284231 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:54:07.285611 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.274565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:33505 (local address 127.4.197.131:34399)
0929 02:54:07.275605 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.275652 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:07.277456 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:07.278202 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:07.278212 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:07.278322 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:07.278988 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.279009 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.280122 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.280127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.281564 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.281578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.282389 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.282396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.282638 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.283369 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:54:07.283371 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:07.283420 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:54:07.283503 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:07.283884 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:07.283895 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:07.283904 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:54:07.284075 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 02:54:07.284496 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":346,"threads_started":1}
I20260929 02:54:07.287070 25210 heartbeater.cc:347] Connected to a master server at 127.4.197.190:33505
I20260929 02:54:07.287413 25210 heartbeater.cc:464] Registering TS with master...
I20260929 02:54:07.288370 25210 heartbeater.cc:511] Master 127.4.197.190:33505 requested a full tablet report, sending...
I20260929 02:54:07.289809 24763 ts_manager.cc:194] Registered new tserver with Master: 5daa3118f34d4b3d938f808c5757ec6b (127.4.197.131:40721)
I20260929 02:54:07.290673 24763 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.197.131:34399
I20260929 02:54:07.300740 4886 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:54:07.309818 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.304667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.4.197.190:33505)
0929 02:54:07.304878 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.304885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.305193 (+ 308us) server_negotiation.cc:407] Connection header received
0929 02:54:07.305326 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.305330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.305404 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.305482 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:07.305842 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.306515 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.307773 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.308178 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.308275 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.309184 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:07.309188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:07.309192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:07.309207 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:07.309237 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:07.309248 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:07.309319 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.309445 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.309531 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:54:07.313259 24763 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:54:07.320225 24763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:54:07.345391 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.332199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42101 (local address 127.0.0.1:39444)
0929 02:54:07.333286 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.333325 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:07.333483 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:07.337445 (+ 3962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:07.337452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:07.337480 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:07.337907 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.337922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.340026 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.340031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.342389 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.342404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.344343 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.344350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.344609 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.344666 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 02:54:07.344800 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:54:07.345009 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":24,"thread_start_us":4356,"threads_started":1}
I20260929 02:54:07.347051 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.332222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.4.197.129:42101)
0929 02:54:07.336735 (+ 4513us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.336744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.336783 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:54:07.336896 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.336904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.337207 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.338125 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:07.338380 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.339880 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.342648 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.344138 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.346144 (+ 2006us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.346409 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.346487 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.346663 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4355,"thread_start_us":4236,"threads_started":1}
I20260929 02:54:07.348102 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.332989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.4.197.130:43783)
0929 02:54:07.336413 (+ 3424us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.336432 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.336961 (+ 529us) server_negotiation.cc:407] Connection header received
0929 02:54:07.337268 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.337277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.337514 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.337692 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:07.339880 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.341430 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.345129 (+ 3699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.346512 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.346611 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.346866 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.347581 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.347749 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3251,"thread_start_us":3127,"threads_started":1}
I20260929 02:54:07.348688 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.332355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.130:43783 (local address 127.0.0.1:53728)
0929 02:54:07.336846 (+ 4491us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.336865 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:07.337067 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:07.338019 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:07.338023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:07.338048 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:07.338442 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.339698 (+ 1256us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.341612 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.341617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.344916 (+ 3299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.344929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.347076 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.347082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.347270 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.347301 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:54:07.347465 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 02:54:07.347652 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4349,"mutex_wait_us":1,"thread_start_us":784,"threads_started":1}
I20260929 02:54:07.351250 24884 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8c607ec4144411b288754de1950d89 (DEFAULT_TABLE table=test-table [id=46d328c219ca4778af413e0d2cae2023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:54:07.352418 24884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8c607ec4144411b288754de1950d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:07.352195 25015 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8c607ec4144411b288754de1950d89 (DEFAULT_TABLE table=test-table [id=46d328c219ca4778af413e0d2cae2023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:54:07.353436 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8c607ec4144411b288754de1950d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:07.354977 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.336943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:40721 (local address 127.0.0.1:60974)
0929 02:54:07.345648 (+ 8705us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.345667 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:07.345776 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:07.348502 (+ 2726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:07.348506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:07.348540 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:07.348738 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.348749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.350831 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.350836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.352587 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.352602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.354245 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.354251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.354431 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.354466 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:54:07.354532 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 02:54:07.354664 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8603,"thread_start_us":9879,"threads_started":1}
I20260929 02:54:07.356266 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.337331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.4.197.131:40721)
0929 02:54:07.347765 (+ 10434us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.347773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.347806 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:54:07.347901 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.347908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.348177 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.348916 (+ 739us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:07.349187 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.350678 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.352747 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.354059 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.355403 (+ 1344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.355656 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.355727 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.355915 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10261,"thread_start_us":11948,"threads_started":1}
I20260929 02:54:07.360165 25145 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8c607ec4144411b288754de1950d89 (DEFAULT_TABLE table=test-table [id=46d328c219ca4778af413e0d2cae2023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:54:07.362016 25145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8c607ec4144411b288754de1950d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:07.365460 25228 tablet_bootstrap.cc:492] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: Bootstrap starting.
I20260929 02:54:07.367508 25229 tablet_bootstrap.cc:492] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0: Bootstrap starting.
I20260929 02:54:07.369069 25228 tablet_bootstrap.cc:654] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:07.370347 25228 log.cc:824] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:07.371151 25229 tablet_bootstrap.cc:654] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:07.372328 25229 log.cc:824] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:07.374346 25228 tablet_bootstrap.cc:492] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: No bootstrap required, opened a new log
I20260929 02:54:07.374662 25228 ts_tablet_manager.cc:1397] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.009s
I20260929 02:54:07.375393 25229 tablet_bootstrap.cc:492] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0: No bootstrap required, opened a new log
I20260929 02:54:07.375602 25232 tablet_bootstrap.cc:492] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: Bootstrap starting.
I20260929 02:54:07.375767 25229 ts_tablet_manager.cc:1397] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260929 02:54:07.378369 25232 tablet_bootstrap.cc:654] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:07.379400 25232 log.cc:824] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:07.382196 25232 tablet_bootstrap.cc:492] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: No bootstrap required, opened a new log
I20260929 02:54:07.382529 25232 ts_tablet_manager.cc:1397] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 02:54:07.385324 25228 raft_consensus.cc:348] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 } } peers { permanent_uuid: "4d7ef05d84834bfbbab9f90628f03b25" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43783 } } peers { permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 } }
I20260929 02:54:07.385792 25228 raft_consensus.cc:374] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:07.385939 25228 raft_consensus.cc:729] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7ef05d84834bfbbab9f90628f03b25, State: Initialized, Role: FOLLOWER
I20260929 02:54:07.385921 25229 raft_consensus.cc:348] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 } } peers { permanent_uuid: "4d7ef05d84834bfbbab9f90628f03b25" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43783 } } peers { permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 } }
I20260929 02:54:07.386368 25229 raft_consensus.cc:374] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:07.386462 25229 raft_consensus.cc:729] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d35aa4c54e04f438c99eabd0c393fa0, State: Initialized, Role: FOLLOWER
I20260929 02:54:07.386606 25228 consensus_queue.cc:260] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 } } peers { permanent_uuid: "4d7ef05d84834bfbbab9f90628f03b25" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43783 } } peers { permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 } }
I20260929 02:54:07.387042 25229 consensus_queue.cc:260] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 } } peers { permanent_uuid: "4d7ef05d84834bfbbab9f90628f03b25" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43783 } } peers { permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 } }
I20260929 02:54:07.387993 25080 heartbeater.cc:503] Master 127.4.197.190:33505 was elected leader, sending a full tablet report...
I20260929 02:54:07.388701 25228 ts_tablet_manager.cc:1428] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260929 02:54:07.389456 24949 heartbeater.cc:503] Master 127.4.197.190:33505 was elected leader, sending a full tablet report...
I20260929 02:54:07.390130 25232 raft_consensus.cc:348] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 } } peers { permanent_uuid: "4d7ef05d84834bfbbab9f90628f03b25" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43783 } } peers { permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 } }
I20260929 02:54:07.390450 25232 raft_consensus.cc:374] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:07.390545 25232 raft_consensus.cc:729] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5daa3118f34d4b3d938f808c5757ec6b, State: Initialized, Role: FOLLOWER
I20260929 02:54:07.390749 25234 raft_consensus.cc:482] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:54:07.391162 25232 consensus_queue.cc:260] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 } } peers { permanent_uuid: "4d7ef05d84834bfbbab9f90628f03b25" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43783 } } peers { permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 } }
I20260929 02:54:07.391820 25229 ts_tablet_manager.cc:1428] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0: Time spent starting tablet: real 0.016s user 0.011s sys 0.001s
I20260929 02:54:07.392688 25234 raft_consensus.cc:504] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 } } peers { permanent_uuid: "4d7ef05d84834bfbbab9f90628f03b25" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43783 } } peers { permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 } }
I20260929 02:54:07.398196 25210 heartbeater.cc:503] Master 127.4.197.190:33505 was elected leader, sending a full tablet report...
I20260929 02:54:07.399120 25232 ts_tablet_manager.cc:1428] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260929 02:54:07.399170 25234 leader_election.cc:288] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d35aa4c54e04f438c99eabd0c393fa0 (127.4.197.129:42101), 5daa3118f34d4b3d938f808c5757ec6b (127.4.197.131:40721)
W20260929 02:54:07.403074 24950 tablet.cc:2369] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:54:07.407793 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.395632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:45429 (local address 127.4.197.129:42101)
0929 02:54:07.395796 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.395800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.400086 (+ 4286us) server_negotiation.cc:407] Connection header received
0929 02:54:07.400379 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.400386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.400494 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.400561 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:07.401398 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.402604 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.405392 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.406696 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.406872 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.406969 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.407389 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.407538 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:54:07.407828 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.395209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42101 (local address 127.4.197.130:45429)
0929 02:54:07.399979 (+ 4770us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.400001 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:07.400204 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:07.400763 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:07.400769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:07.400795 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:07.401230 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.401246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.402789 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.402795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.405184 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.405199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.406902 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.406909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.407197 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.407238 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 02:54:07.407349 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:54:07.407479 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4594,"thread_start_us":4390,"threads_started":1}
I20260929 02:54:07.411204 24904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8c607ec4144411b288754de1950d89" candidate_uuid: "4d7ef05d84834bfbbab9f90628f03b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" is_pre_election: true
I20260929 02:54:07.411692 24904 raft_consensus.cc:2454] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7ef05d84834bfbbab9f90628f03b25 in term 0.
I20260929 02:54:07.412310 24968 leader_election.cc:302] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d35aa4c54e04f438c99eabd0c393fa0, 4d7ef05d84834bfbbab9f90628f03b25; no voters:
I20260929 02:54:07.412655 25234 raft_consensus.cc:2781] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:54:07.412778 25234 raft_consensus.cc:482] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:54:07.412876 25234 raft_consensus.cc:3037] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:07.415699 25234 raft_consensus.cc:504] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 } } peers { permanent_uuid: "4d7ef05d84834bfbbab9f90628f03b25" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43783 } } peers { permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 } }
I20260929 02:54:07.415894 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.400146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.197.130:45245 (local address 127.4.197.131:40721)
0929 02:54:07.400316 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.400322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.409109 (+ 8787us) server_negotiation.cc:407] Connection header received
0929 02:54:07.409363 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.409370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.409450 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.409511 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:54:07.410468 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.411502 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.413831 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.415003 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.415142 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.415206 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.415558 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.415681 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:54:07.416002 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.399979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.131:40721 (local address 127.4.197.130:45245)
0929 02:54:07.408046 (+ 8067us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.408065 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:07.409209 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:07.409684 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:07.409689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:07.409725 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:54:07.409889 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.409900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.411699 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.411705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.413638 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:07.413658 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.415158 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.415165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:07.415374 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.415419 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 02:54:07.415517 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:54:07.415654 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7955,"thread_start_us":10218,"threads_started":1}
I20260929 02:54:07.416342 25234 leader_election.cc:288] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [CANDIDATE]: Term 1 election: Requested vote from peers 2d35aa4c54e04f438c99eabd0c393fa0 (127.4.197.129:42101), 5daa3118f34d4b3d938f808c5757ec6b (127.4.197.131:40721)
I20260929 02:54:07.416951 25165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8c607ec4144411b288754de1950d89" candidate_uuid: "4d7ef05d84834bfbbab9f90628f03b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5daa3118f34d4b3d938f808c5757ec6b" is_pre_election: true
I20260929 02:54:07.417076 24904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8c607ec4144411b288754de1950d89" candidate_uuid: "4d7ef05d84834bfbbab9f90628f03b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d35aa4c54e04f438c99eabd0c393fa0"
I20260929 02:54:07.417282 24904 raft_consensus.cc:3037] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:07.417253 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8c607ec4144411b288754de1950d89" candidate_uuid: "4d7ef05d84834bfbbab9f90628f03b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5daa3118f34d4b3d938f808c5757ec6b"
I20260929 02:54:07.417457 25165 raft_consensus.cc:2454] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7ef05d84834bfbbab9f90628f03b25 in term 0.
I20260929 02:54:07.419706 24904 raft_consensus.cc:2454] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7ef05d84834bfbbab9f90628f03b25 in term 1.
I20260929 02:54:07.420048 24968 leader_election.cc:302] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d35aa4c54e04f438c99eabd0c393fa0, 4d7ef05d84834bfbbab9f90628f03b25; no voters: 5daa3118f34d4b3d938f808c5757ec6b
I20260929 02:54:07.420339 25234 raft_consensus.cc:2781] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:07.420753 25234 raft_consensus.cc:686] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 1 LEADER]: Becoming Leader. State: Replica: 4d7ef05d84834bfbbab9f90628f03b25, State: Running, Role: LEADER
I20260929 02:54:07.421240 25234 consensus_queue.cc:237] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 } } peers { permanent_uuid: "4d7ef05d84834bfbbab9f90628f03b25" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43783 } } peers { permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 } }
I20260929 02:54:07.425542 24763 catalog_manager.cc:5495] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d7ef05d84834bfbbab9f90628f03b25 (127.4.197.130). New cstate: current_term: 1 leader_uuid: "4d7ef05d84834bfbbab9f90628f03b25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d7ef05d84834bfbbab9f90628f03b25" member_type: VOTER last_known_addr { host: "127.4.197.130" port: 43783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 } health_report { overall_health: UNKNOWN } } }
I20260929 02:54:07.441987 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.436589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.4.197.130:43783)
0929 02:54:07.436779 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.436785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.436907 (+ 122us) server_negotiation.cc:407] Connection header received
0929 02:54:07.437126 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.437130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.437222 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.437359 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:07.437754 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.438473 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.439925 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.440330 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.440440 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.440909 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:07.441428 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:07.441561 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.441656 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.441738 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:54:07.449693 25164 raft_consensus.cc:3037] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:07.449949 24904 raft_consensus.cc:1261] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0 [term 1 FOLLOWER]: Refusing update from remote peer 4d7ef05d84834bfbbab9f90628f03b25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:54:07.450722 25239 consensus_queue.cc:1035] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d35aa4c54e04f438c99eabd0c393fa0" member_type: VOTER last_known_addr { host: "127.4.197.129" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:54:07.452495 25164 raft_consensus.cc:1261] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b [term 1 FOLLOWER]: Refusing update from remote peer 4d7ef05d84834bfbbab9f90628f03b25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:54:07.453262 25239 consensus_queue.cc:1035] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5daa3118f34d4b3d938f808c5757ec6b" member_type: VOTER last_known_addr { host: "127.4.197.131" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:54:07.462198 25243 mvcc.cc:205] Tried to move back new op lower bound from 7334504232744919040 to 7334504232641097728. Current Snapshot: MvccSnapshot[applied={T|T < 7334504232744919040}]
I20260929 02:54:07.479681 24762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:54:07.480005 24762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:54:07.484298 24762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07fb2ab0d90b49fe834309c1e5b4f7d7: Sending DeleteTablet for 3 replicas of tablet 1d8c607ec4144411b288754de1950d89
I20260929 02:54:07.485589 25145 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8c607ec4144411b288754de1950d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:54:07 UTC) from {username='slave'} at 127.0.0.1:60974
I20260929 02:54:07.485739 25015 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8c607ec4144411b288754de1950d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53728
I20260929 02:54:07.485749 24884 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8c607ec4144411b288754de1950d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:54:07 UTC) from {username='slave'} at 127.0.0.1:39444
I20260929 02:54:07.486320 25014 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8c607ec4144411b288754de1950d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53728
I20260929 02:54:07.486527 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 24822
I20260929 02:54:07.496755 25253 tablet_replica.cc:330] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0: stopping tablet replica
I20260929 02:54:07.497346 25253 raft_consensus.cc:2229] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:54:07.498148 25253 raft_consensus.cc:2258] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:54:07.498708 25252 tablet_replica.cc:330] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: stopping tablet replica
I20260929 02:54:07.499825 25252 raft_consensus.cc:2229] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:54:07.501398 25253 ts_tablet_manager.cc:1905] T 1d8c607ec4144411b288754de1950d89 P 2d35aa4c54e04f438c99eabd0c393fa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:54:07.501905 25252 raft_consensus.cc:2258] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:54:07.504956 25254 tablet_replica.cc:330] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: stopping tablet replica
I20260929 02:54:07.505277 25254 raft_consensus.cc:2229] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:54:07.505777 25254 raft_consensus.cc:2258] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:54:07.507838 25254 ts_tablet_manager.cc:1905] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:54:07.508706 25252 ts_tablet_manager.cc:1905] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:54:07.511679 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.487943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.4.197.129:42101)
0929 02:54:07.488129 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.488135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.498209 (+ 10074us) server_negotiation.cc:407] Connection header received
0929 02:54:07.498695 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.498702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.498829 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.498907 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:07.499605 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.500836 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.502832 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.503581 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.503686 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.506384 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:07.506396 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:07.506412 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:07.507197 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:07.507254 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:07.507279 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:07.507384 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.511217 (+ 3833us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.511328 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:54:07.512948 24944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:07.514329 25254 ts_tablet_manager.cc:1918] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:54:07.514551 25254 log.cc:1196] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-1/wal/wals/1d8c607ec4144411b288754de1950d89
I20260929 02:54:07.515193 25252 ts_tablet_manager.cc:1918] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:54:07.515242 25254 ts_tablet_manager.cc:1939] T 1d8c607ec4144411b288754de1950d89 P 4d7ef05d84834bfbbab9f90628f03b25: Deleting consensus metadata
I20260929 02:54:07.515332 25252 log.cc:1196] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: Deleting WAL directory at /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650137986910-4886-0/minicluster-data/ts-2/wal/wals/1d8c607ec4144411b288754de1950d89
I20260929 02:54:07.515808 25252 ts_tablet_manager.cc:1939] T 1d8c607ec4144411b288754de1950d89 P 5daa3118f34d4b3d938f808c5757ec6b: Deleting consensus metadata
I20260929 02:54:07.517294 24749 catalog_manager.cc:4841] TS 4d7ef05d84834bfbbab9f90628f03b25 (127.4.197.130:43783): tablet 1d8c607ec4144411b288754de1950d89 (table test-table [id=46d328c219ca4778af413e0d2cae2023]) successfully deleted
I20260929 02:54:07.517509 24747 catalog_manager.cc:4841] TS 5daa3118f34d4b3d938f808c5757ec6b (127.4.197.131:40721): tablet 1d8c607ec4144411b288754de1950d89 (table test-table [id=46d328c219ca4778af413e0d2cae2023]) successfully deleted
W20260929 02:54:07.517989 24749 catalog_manager.cc:4808] TS 4d7ef05d84834bfbbab9f90628f03b25 (127.4.197.130:43783): delete failed for tablet 1d8c607ec4144411b288754de1950d89 because the tablet was not found. No further retry: Not found: Tablet not found: 1d8c607ec4144411b288754de1950d89
I20260929 02:54:07.606310 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 24822
W20260929 02:54:07.627326 24747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.197.129:42101 (error 108)
I20260929 02:54:07.628453 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 24953
I20260929 02:54:07.629247 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.628747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42101 (local address 127.0.0.1:39454)
0929 02:54:07.628947 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.629055 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 02:54:07.629472 24747 catalog_manager.cc:4584] TS 2d35aa4c54e04f438c99eabd0c393fa0 (127.4.197.129:42101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d8c607ec4144411b288754de1950d89: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42101: connect: Connection refused (error 111)
I20260929 02:54:07.635337 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.629252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.4.197.130:43783)
0929 02:54:07.629675 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.629681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.629698 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:54:07.629756 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.629760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.629857 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.629942 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:07.630376 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.631175 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.632426 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.632848 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.632957 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.633996 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:07.634006 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:07.634020 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:07.634612 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:07.634662 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:07.634685 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:07.634775 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.634910 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.635034 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260929 02:54:07.636036 25075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:07.683103 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.682458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42101 (local address 127.0.0.1:57290)
0929 02:54:07.682721 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.682845 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 02:54:07.684264 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.683855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42101 (local address 127.0.0.1:57302)
0929 02:54:07.684013 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.684101 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 02:54:07.744967 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.744388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42101 (local address 127.0.0.1:57312)
0929 02:54:07.744634 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.744753 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:54:07.746227 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.745768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42101 (local address 127.0.0.1:57316)
0929 02:54:07.745938 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.746034 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 02:54:07.750200 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 24953
I20260929 02:54:07.765342 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25083
I20260929 02:54:07.775264 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.766056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.4.197.131:40721)
0929 02:54:07.766223 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:07.766228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:07.766244 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:54:07.766402 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:07.766406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:07.766485 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:07.766563 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:07.766891 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.767635 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.768803 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:07.769188 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:07.769269 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:07.773839 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:07.773848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:07.773863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:07.774476 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:07.774527 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:07.774550 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:07.774641 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:07.774796 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 02:54:07.774903 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:54:07.775914 25205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:07.826655 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.826224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42101 (local address 127.0.0.1:57330)
0929 02:54:07.826412 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.826495 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 02:54:07.827723 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:07.827306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.129:42101 (local address 127.0.0.1:57346)
0929 02:54:07.827518 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:07.827593 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.129:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 02:54:07.875219 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 25083
I20260929 02:54:07.890201 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 24732
I20260929 02:54:07.891295 24793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:07.998819 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 24732
2026-09-29T02:54:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2654 ms)
[----------] 6 tests from EncryptionPolicyTest (20814 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:08 dist-test-slave-t6c1 krb5kdc[25263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:08 dist-test-slave-t6c1 krb5kdc[25263](info): set up 2 sockets
Sep 29 02:54:08 dist-test-slave-t6c1 krb5kdc[25263](info): commencing operation
krb5kdc: starting...
W20260929 02:54:09.007313 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:09 dist-test-slave-t6c1 krb5kdc[25263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:09.234876 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:09.552554 25276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:09.552855 25276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:09.552996 25276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:09.560364 25276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:09.560492 25276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:09.560554 25276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:09.560634 25276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:09.560700 25276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:09.570681 25276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:09.572043 25276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:09.573392 25276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:09.587224 25281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:09.587188 25282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:09.587157 25284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:09.588380 25276 server_base.cc:1034] running on GCE node
I20260929 02:54:09.589180 25276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:09.590565 25276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:09.591754 25276 hybrid_clock.cc:648] HybridClock initialized: now 1790650449591701 us; error 49 us; skew 500 ppm
I20260929 02:54:09.592087 25276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:09 dist-test-slave-t6c1 krb5kdc[25263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:09.594991 25276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:09.596447 25276 webserver.cc:460] Webserver started at http://127.0.0.1:41099/ using document root <none> and password file <none>
I20260929 02:54:09.597096 25276 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:09.597204 25276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:09.597471 25276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:09.600342 25276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "bd3798830d0e46b18da9a43fc1c3f3be"
format_stamp: "Formatted at 2026-09-29 02:54:09 on dist-test-slave-t6c1"
server_key: "5eb54e9e2001b130523079459ea8e43f"
server_key_iv: "a27d6d3f8a95fddc1d197619a512a57d"
server_key_version: "encryptionkey@0"
I20260929 02:54:09.601058 25276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "bd3798830d0e46b18da9a43fc1c3f3be"
format_stamp: "Formatted at 2026-09-29 02:54:09 on dist-test-slave-t6c1"
server_key: "5eb54e9e2001b130523079459ea8e43f"
server_key_iv: "a27d6d3f8a95fddc1d197619a512a57d"
server_key_version: "encryptionkey@0"
I20260929 02:54:09.606178 25276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:54:09.609587 25291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:09.610827 25276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:54:09.611014 25276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "bd3798830d0e46b18da9a43fc1c3f3be"
format_stamp: "Formatted at 2026-09-29 02:54:09 on dist-test-slave-t6c1"
server_key: "5eb54e9e2001b130523079459ea8e43f"
server_key_iv: "a27d6d3f8a95fddc1d197619a512a57d"
server_key_version: "encryptionkey@0"
I20260929 02:54:09.611193 25276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:09.643224 25276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:09.647497 25276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:09.647774 25276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:09.664897 25276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33789
I20260929 02:54:09.664918 25342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33789 every 8 connection(s)
I20260929 02:54:09.666373 25276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:09.668159 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 25276
I20260929 02:54:09.668355 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:09.668955 4886 external_mini_cluster.cc:1428] Setting key 749f64b40a2b9b1a781a536fb482ce15
I20260929 02:54:09.670754 25343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:54:09 dist-test-slave-t6c1 krb5kdc[25263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:54:09.684356 25343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be: Bootstrap starting.
I20260929 02:54:09.688444 25343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:09.689643 25343 log.cc:824] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:09.692884 25343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be: No bootstrap required, opened a new log
I20260929 02:54:09.692983 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:09.671007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33137 (local address 127.0.0.1:33789)
0929 02:54:09.671817 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:09.671835 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:09.671889 (+ 54us) server_negotiation.cc:407] Connection header received
0929 02:54:09.674001 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:09.674031 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:09.674606 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:09.675122 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:09.676376 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:09.678394 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:09.679646 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:09.680658 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:09.680840 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:09.684895 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:09.684905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:09.684935 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:09.684978 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:09.688627 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:09.689383 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:09.689391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:09.689400 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:09.689606 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:09.690013 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:09.690018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:09.690020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:09.690611 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:09.690848 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:09.691403 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 02:54:09.691718 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":326,"threads_started":1}
I20260929 02:54:09.696779 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:09.700793 25343 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3798830d0e46b18da9a43fc1c3f3be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260929 02:54:09.701200 25343 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:09.701320 25343 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3798830d0e46b18da9a43fc1c3f3be, State: Initialized, Role: FOLLOWER
I20260929 02:54:09.702054 25343 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3798830d0e46b18da9a43fc1c3f3be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260929 02:54:09.702338 25343 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:09.702491 25343 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:09.702705 25343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:09.705536 25343 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3798830d0e46b18da9a43fc1c3f3be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260929 02:54:09.706043 25343 leader_election.cc:302] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd3798830d0e46b18da9a43fc1c3f3be; no voters:
I20260929 02:54:09.706681 25343 leader_election.cc:288] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:09.706805 25353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:09.708351 25353 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [term 1 LEADER]: Becoming Leader. State: Replica: bd3798830d0e46b18da9a43fc1c3f3be, State: Running, Role: LEADER
I20260929 02:54:09.708912 25353 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3798830d0e46b18da9a43fc1c3f3be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260929 02:54:09.709167 25343 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:09.711774 25355 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd3798830d0e46b18da9a43fc1c3f3be. Latest consensus state: current_term: 1 leader_uuid: "bd3798830d0e46b18da9a43fc1c3f3be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3798830d0e46b18da9a43fc1c3f3be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } } }
I20260929 02:54:09.712141 25355 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:09.713891 25354 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd3798830d0e46b18da9a43fc1c3f3be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3798830d0e46b18da9a43fc1c3f3be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } } }
I20260929 02:54:09.714233 25354 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:09.715785 25359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:09.718743 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:09.701361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36533 (local address 127.0.0.1:33789)
0929 02:54:09.701533 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:09.701539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:09.702219 (+ 680us) server_negotiation.cc:407] Connection header received
0929 02:54:09.702414 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:09.702419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:09.702508 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:09.702613 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:09.704150 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:09.705285 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:09.706571 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:09.707281 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:09.707426 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:09.710062 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:09.710068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:09.710074 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:09.710109 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:09.715595 (+ 5486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:09.716408 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:09.716414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:09.716418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:09.716631 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:09.717123 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:09.717128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:09.717132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:09.717512 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:09.717819 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:09.718192 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 02:54:09.718378 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:54:09.726012 25359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:09.745872 25359 catalog_manager.cc:1347] Generated new cluster ID: 4cc6b8c1d4d940f6b7929cc2d79f4c78
I20260929 02:54:09.746181 25359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:09.762715 25359 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:09.764201 25359 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:09.774562 25359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd3798830d0e46b18da9a43fc1c3f3be: Generated new TSK 0
I20260929 02:54:09.775465 25359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:09.780927 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25276
I20260929 02:54:09.782635 25338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:09.874313 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 25276
W20260929 02:54:09.892146 25261 connection.cc:535] client connection to 127.0.0.1:33789 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33789): Connection reset by peer (error 104)
2026-09-29T02:54:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:09 dist-test-slave-t6c1 krb5kdc[25375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:09 dist-test-slave-t6c1 krb5kdc[25375](info): set up 2 sockets
Sep 29 02:54:09 dist-test-slave-t6c1 krb5kdc[25375](info): commencing operation
krb5kdc: starting...
W20260929 02:54:10.891183 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:11 dist-test-slave-t6c1 krb5kdc[25375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:11.131712 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:11.467209 25388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:11.467518 25388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:11.467646 25388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:11.475075 25388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:11.475198 25388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:11.475270 25388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:11.475340 25388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:11.475384 25388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:11.486693 25388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:11.488083 25388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:11.489406 25388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:11.501142 25394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:11.501168 25396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:11.501266 25393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:11.502003 25388 server_base.cc:1034] running on GCE node
I20260929 02:54:11.502847 25388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:11.504382 25388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:11.505594 25388 hybrid_clock.cc:648] HybridClock initialized: now 1790650451505538 us; error 45 us; skew 500 ppm
I20260929 02:54:11.505966 25388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:11 dist-test-slave-t6c1 krb5kdc[25375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:11.508975 25388 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:11.510453 25388 webserver.cc:460] Webserver started at http://127.0.0.1:43161/ using document root <none> and password file <none>
I20260929 02:54:11.511157 25388 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:11.511262 25388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:11.511519 25388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:11.514400 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "abc80725ebb64a6292c0d152a4d1d9b6"
format_stamp: "Formatted at 2026-09-29 02:54:11 on dist-test-slave-t6c1"
server_key: "c9073acd1dd109c1f83671d19af5bde0"
server_key_iv: "d69c16432c28eba3e91cd6c87a2b2d5d"
server_key_version: "encryptionkey@0"
I20260929 02:54:11.515189 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "abc80725ebb64a6292c0d152a4d1d9b6"
format_stamp: "Formatted at 2026-09-29 02:54:11 on dist-test-slave-t6c1"
server_key: "c9073acd1dd109c1f83671d19af5bde0"
server_key_iv: "d69c16432c28eba3e91cd6c87a2b2d5d"
server_key_version: "encryptionkey@0"
I20260929 02:54:11.519938 25388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:54:11.523100 25403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:11.524570 25388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:54:11.524708 25388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "abc80725ebb64a6292c0d152a4d1d9b6"
format_stamp: "Formatted at 2026-09-29 02:54:11 on dist-test-slave-t6c1"
server_key: "c9073acd1dd109c1f83671d19af5bde0"
server_key_iv: "d69c16432c28eba3e91cd6c87a2b2d5d"
server_key_version: "encryptionkey@0"
I20260929 02:54:11.524883 25388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:11.549528 25388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:11.553407 25388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:11.553654 25388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:11.570392 25388 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38629
I20260929 02:54:11.570405 25454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38629 every 8 connection(s)
I20260929 02:54:11.571768 25388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:11.574484 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 25388
I20260929 02:54:11.574626 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:11.575124 4886 external_mini_cluster.cc:1428] Setting key e32d10e737fb23ebd21c5bfbb0df97ca
I20260929 02:54:11.575518 25455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:11.591020 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:11.576864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35955 (local address 127.0.0.1:38629)
0929 02:54:11.577728 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:11.577743 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:11.577792 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:54:11.580305 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:11.580337 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:11.580933 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:11.581420 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:11.582001 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:11.584121 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:11.585531 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:11.586597 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:11.586785 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:11.587948 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:11.587960 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:11.587983 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:11.588788 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:11.588890 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:11.588929 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:11.589088 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:11.589303 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 02:54:11.589525 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":322,"threads_started":1}
I20260929 02:54:11.591105 25455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6: Bootstrap starting.
I20260929 02:54:11.594493 25455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:11.595073 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:11.595660 25455 log.cc:824] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:11.598693 25455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6: No bootstrap required, opened a new log
I20260929 02:54:11.605759 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:11.599667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49323 (local address 127.0.0.1:38629)
0929 02:54:11.599825 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:11.599832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:11.600236 (+ 404us) server_negotiation.cc:407] Connection header received
0929 02:54:11.600436 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:11.600440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:11.600512 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:11.600611 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:11.601123 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:11.601919 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:11.603251 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:11.603730 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:11.603848 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:11.605062 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:11.605067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:11.605070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:11.605090 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:11.605131 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:11.605150 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:11.605241 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:11.605446 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 02:54:11.605499 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:54:11.607309 25455 raft_consensus.cc:348] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc80725ebb64a6292c0d152a4d1d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260929 02:54:11.607682 25455 raft_consensus.cc:374] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:11.607775 25455 raft_consensus.cc:729] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc80725ebb64a6292c0d152a4d1d9b6, State: Initialized, Role: FOLLOWER
I20260929 02:54:11.608366 25455 consensus_queue.cc:260] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc80725ebb64a6292c0d152a4d1d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260929 02:54:11.608575 25455 raft_consensus.cc:388] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:11.608705 25455 raft_consensus.cc:482] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:11.608841 25455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:11.611296 25455 raft_consensus.cc:504] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc80725ebb64a6292c0d152a4d1d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260929 02:54:11.611764 25455 leader_election.cc:302] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abc80725ebb64a6292c0d152a4d1d9b6; no voters:
I20260929 02:54:11.612322 25455 leader_election.cc:288] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:11.612432 25465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:11.613065 25465 raft_consensus.cc:686] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [term 1 LEADER]: Becoming Leader. State: Replica: abc80725ebb64a6292c0d152a4d1d9b6, State: Running, Role: LEADER
I20260929 02:54:11.613538 25465 consensus_queue.cc:237] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc80725ebb64a6292c0d152a4d1d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260929 02:54:11.614282 25455 sys_catalog.cc:564] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:11.616163 25466 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abc80725ebb64a6292c0d152a4d1d9b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc80725ebb64a6292c0d152a4d1d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } } }
I20260929 02:54:11.616168 25467 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abc80725ebb64a6292c0d152a4d1d9b6. Latest consensus state: current_term: 1 leader_uuid: "abc80725ebb64a6292c0d152a4d1d9b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc80725ebb64a6292c0d152a4d1d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } } }
I20260929 02:54:11.616612 25466 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:11.618665 25467 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:11.619575 25470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:11.624411 25470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:11.634076 25470 catalog_manager.cc:1347] Generated new cluster ID: 6ac25cd752744ac18bdcaf6cec630003
I20260929 02:54:11.634214 25470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:11.664125 25470 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:11.665376 25470 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:11.680954 25470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abc80725ebb64a6292c0d152a4d1d9b6: Generated new TSK 0
I20260929 02:54:11.682113 25470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:11.691916 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25388
W20260929 02:54:11.693109 25450 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35955
W20260929 02:54:11.693847 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25388: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:11.694000 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 25388
W20260929 02:54:11.707217 25373 connection.cc:535] client connection to 127.0.0.1:38629 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38629): Connection reset by peer (error 104)
2026-09-29T02:54:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1814 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:11 dist-test-slave-t6c1 krb5kdc[25486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:11 dist-test-slave-t6c1 krb5kdc[25486](info): set up 2 sockets
Sep 29 02:54:11 dist-test-slave-t6c1 krb5kdc[25486](info): commencing operation
krb5kdc: starting...
W20260929 02:54:12.699069 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:12 dist-test-slave-t6c1 krb5kdc[25486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:12.934121 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:13.251662 25499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:13.251960 25499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:13.252067 25499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:13.259466 25499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:13.259579 25499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:13.259631 25499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:13.259689 25499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:13.259732 25499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:13.269672 25499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:13.270939 25499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:13.272239 25499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:13.283989 25504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:13.284055 25505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:13.284029 25507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:13.284811 25499 server_base.cc:1034] running on GCE node
I20260929 02:54:13.285629 25499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:13.287053 25499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:13.288254 25499 hybrid_clock.cc:648] HybridClock initialized: now 1790650453288202 us; error 43 us; skew 500 ppm
I20260929 02:54:13.288590 25499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:13 dist-test-slave-t6c1 krb5kdc[25486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:13.294770 25499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:13.296288 25499 webserver.cc:460] Webserver started at http://127.0.0.1:35359/ using document root <none> and password file <none>
I20260929 02:54:13.296932 25499 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:13.297055 25499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:13.297326 25499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:13.300174 25499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d"
format_stamp: "Formatted at 2026-09-29 02:54:13 on dist-test-slave-t6c1"
server_key: "373bf8385186d6ae557cfe7a9a1febac"
server_key_iv: "dcda6dc4745d449d634a01809020abcd"
server_key_version: "encryptionkey@0"
I20260929 02:54:13.300863 25499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d"
format_stamp: "Formatted at 2026-09-29 02:54:13 on dist-test-slave-t6c1"
server_key: "373bf8385186d6ae557cfe7a9a1febac"
server_key_iv: "dcda6dc4745d449d634a01809020abcd"
server_key_version: "encryptionkey@0"
I20260929 02:54:13.305644 25499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 02:54:13.308836 25514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:13.310073 25499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:54:13.310230 25499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d"
format_stamp: "Formatted at 2026-09-29 02:54:13 on dist-test-slave-t6c1"
server_key: "373bf8385186d6ae557cfe7a9a1febac"
server_key_iv: "dcda6dc4745d449d634a01809020abcd"
server_key_version: "encryptionkey@0"
I20260929 02:54:13.310408 25499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:13.315779 25499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:13.319713 25499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:13.319971 25499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:13.336807 25499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39711
I20260929 02:54:13.336839 25565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39711 every 8 connection(s)
I20260929 02:54:13.338264 25499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:13.342015 25566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:13.346532 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 25499
I20260929 02:54:13.346740 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:13.347326 4886 external_mini_cluster.cc:1428] Setting key 1d11d2127bacfc847f56d450b035c186
I20260929 02:54:13.353637 25566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d: Bootstrap starting.
I20260929 02:54:13.355679 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:13.349263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36361 (local address 127.0.0.1:39711)
0929 02:54:13.350053 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:13.350066 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:13.350107 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:54:13.352260 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:13.352286 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:13.352661 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:13.353025 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:13.354043 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:13.354049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:13.354061 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:13.354076 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:13.354145 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:13.354164 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:13.354362 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:13.354506 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 02:54:13.354682 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":339,"threads_started":1}
I20260929 02:54:13.357149 25566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:13.358304 25566 log.cc:824] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:13.359444 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:13.361509 25566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d: No bootstrap required, opened a new log
I20260929 02:54:13.366807 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:13.364490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42059 (local address 127.0.0.1:39711)
0929 02:54:13.364670 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:13.364676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:13.364969 (+ 293us) server_negotiation.cc:407] Connection header received
0929 02:54:13.365116 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:13.365125 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:13.365190 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:13.365306 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:13.366179 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:13.366184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:13.366188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:13.366207 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:13.366257 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:13.366278 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:13.366408 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:13.366459 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 02:54:13.366518 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:54:13.370867 25566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260929 02:54:13.371392 25566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:13.371520 25566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 876f9dd1c96844c68a5fd8b5bc88aa2d, State: Initialized, Role: FOLLOWER
I20260929 02:54:13.372117 25566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260929 02:54:13.372325 25566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:13.372422 25566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:13.372555 25566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:13.374663 25566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260929 02:54:13.375209 25566 leader_election.cc:302] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 876f9dd1c96844c68a5fd8b5bc88aa2d; no voters:
I20260929 02:54:13.375725 25566 leader_election.cc:288] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:13.375880 25576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:13.376916 25576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [term 1 LEADER]: Becoming Leader. State: Replica: 876f9dd1c96844c68a5fd8b5bc88aa2d, State: Running, Role: LEADER
I20260929 02:54:13.377497 25576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260929 02:54:13.377812 25566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:13.380452 25577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } } }
I20260929 02:54:13.380882 25577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:13.380765 25578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 876f9dd1c96844c68a5fd8b5bc88aa2d. Latest consensus state: current_term: 1 leader_uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876f9dd1c96844c68a5fd8b5bc88aa2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } } }
I20260929 02:54:13.381039 25578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:13.383108 25584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:13.387590 25584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:13.396059 25584 catalog_manager.cc:1347] Generated new cluster ID: 27a77dae11334a96899164469f4e6fdc
I20260929 02:54:13.396174 25584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:13.407202 25584 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:13.407554 25584 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:13.415414 25584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 876f9dd1c96844c68a5fd8b5bc88aa2d: Generated new TSK 0
I20260929 02:54:13.416115 25584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:13.426071 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25499
W20260929 02:54:13.427747 25561 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36361
W20260929 02:54:13.428375 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25499: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:13.428524 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 25499
2026-09-29T02:54:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1734 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:13 dist-test-slave-t6c1 krb5kdc[25597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:13 dist-test-slave-t6c1 krb5kdc[25597](info): set up 2 sockets
Sep 29 02:54:13 dist-test-slave-t6c1 krb5kdc[25597](info): commencing operation
krb5kdc: starting...
W20260929 02:54:14.431166 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:14 dist-test-slave-t6c1 krb5kdc[25597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:14.667672 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:14.987356 25610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:14.987648 25610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:14.987771 25610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:14.995098 25610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:14.995224 25610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:14.995301 25610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:14.995383 25610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:14.995438 25610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:15.005400 25610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:15.006717 25610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:15.008059 25610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:15.019307 25618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:15.019363 25616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:15.019341 25615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:15.020370 25610 server_base.cc:1034] running on GCE node
I20260929 02:54:15.021123 25610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:15.022503 25610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:15.023715 25610 hybrid_clock.cc:648] HybridClock initialized: now 1790650455023664 us; error 40 us; skew 500 ppm
I20260929 02:54:15.024060 25610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:15 dist-test-slave-t6c1 krb5kdc[25597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:15.027113 25610 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:15.028620 25610 webserver.cc:460] Webserver started at http://127.0.0.1:40639/ using document root <none> and password file <none>
I20260929 02:54:15.029333 25610 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:15.029452 25610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:15.029748 25610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:15.032869 25610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "6226c4cf0dfd4c31b88444f0365b8b6d"
format_stamp: "Formatted at 2026-09-29 02:54:15 on dist-test-slave-t6c1"
server_key: "2d346b21158e07494bcf232a5d07fb4a"
server_key_iv: "2fdb7ff08ba272ee115c878cf4c2bd97"
server_key_version: "encryptionkey@0"
I20260929 02:54:15.033674 25610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "6226c4cf0dfd4c31b88444f0365b8b6d"
format_stamp: "Formatted at 2026-09-29 02:54:15 on dist-test-slave-t6c1"
server_key: "2d346b21158e07494bcf232a5d07fb4a"
server_key_iv: "2fdb7ff08ba272ee115c878cf4c2bd97"
server_key_version: "encryptionkey@0"
I20260929 02:54:15.039001 25610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:54:15.042683 25625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:15.044185 25610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:54:15.044361 25610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "6226c4cf0dfd4c31b88444f0365b8b6d"
format_stamp: "Formatted at 2026-09-29 02:54:15 on dist-test-slave-t6c1"
server_key: "2d346b21158e07494bcf232a5d07fb4a"
server_key_iv: "2fdb7ff08ba272ee115c878cf4c2bd97"
server_key_version: "encryptionkey@0"
I20260929 02:54:15.044555 25610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:15.062625 25610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:15.066771 25610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:15.067085 25610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:15.083741 25610 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40497
I20260929 02:54:15.083763 25676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40497 every 8 connection(s)
I20260929 02:54:15.085152 25610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:15.088910 25677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:15.090070 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 25610
I20260929 02:54:15.090245 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:15.090663 4886 external_mini_cluster.cc:1428] Setting key 071e410b3fa42d6361e50900772dd160
I20260929 02:54:15.102177 25677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d: Bootstrap starting.
Sep 29 02:54:15 dist-test-slave-t6c1 krb5kdc[25597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:54:15.105899 25677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:15.107056 25677 log.cc:824] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:15.110114 25677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d: No bootstrap required, opened a new log
I20260929 02:54:15.112974 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:15.092381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39453 (local address 127.0.0.1:40497)
0929 02:54:15.093162 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:15.093173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:15.093212 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:54:15.094809 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:15.094831 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:15.095297 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:15.095622 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:15.097312 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:15.098579 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:15.100317 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:15.100954 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:15.101089 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:15.105833 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:15.105840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:15.105854 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:15.105895 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:15.109148 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:15.109893 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:15.109899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:15.109905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:15.110069 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:15.110434 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:15.110438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:15.110440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:15.110976 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:15.111211 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:15.111612 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 02:54:15.111950 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":340,"threads_started":1}
I20260929 02:54:15.116619 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:15.119364 25677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6226c4cf0dfd4c31b88444f0365b8b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260929 02:54:15.119796 25677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:15.119895 25677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6226c4cf0dfd4c31b88444f0365b8b6d, State: Initialized, Role: FOLLOWER
I20260929 02:54:15.120553 25677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6226c4cf0dfd4c31b88444f0365b8b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260929 02:54:15.120814 25677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:15.120944 25677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:15.121105 25677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:15.124177 25677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6226c4cf0dfd4c31b88444f0365b8b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260929 02:54:15.124671 25677 leader_election.cc:302] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6226c4cf0dfd4c31b88444f0365b8b6d; no voters:
I20260929 02:54:15.125222 25677 leader_election.cc:288] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:15.125342 25687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:15.127269 25677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:15.131217 25687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [term 1 LEADER]: Becoming Leader. State: Replica: 6226c4cf0dfd4c31b88444f0365b8b6d, State: Running, Role: LEADER
I20260929 02:54:15.132042 25687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6226c4cf0dfd4c31b88444f0365b8b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260929 02:54:15.135175 25689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6226c4cf0dfd4c31b88444f0365b8b6d. Latest consensus state: current_term: 1 leader_uuid: "6226c4cf0dfd4c31b88444f0365b8b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6226c4cf0dfd4c31b88444f0365b8b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260929 02:54:15.135572 25689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:15.138948 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:15.121631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53961 (local address 127.0.0.1:40497)
0929 02:54:15.121899 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:15.121903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:15.122196 (+ 293us) server_negotiation.cc:407] Connection header received
0929 02:54:15.122358 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:15.122361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:15.122419 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:15.122495 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:15.124198 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:15.125056 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:15.126679 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:15.127505 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:15.127642 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:15.130772 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:15.130780 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:15.130785 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:15.130828 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:15.134573 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:15.135453 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:15.135458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:15.135462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:15.135613 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:15.136167 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:15.136172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:15.136176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:15.136630 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:15.138124 (+ 1494us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:15.138285 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 02:54:15.138528 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260929 02:54:15.140938 25688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6226c4cf0dfd4c31b88444f0365b8b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6226c4cf0dfd4c31b88444f0365b8b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260929 02:54:15.141239 25688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:15.141628 25702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:15.146340 25702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:15.155507 25702 catalog_manager.cc:1347] Generated new cluster ID: 6ed6657fd1324bffa45c2696322bff16
I20260929 02:54:15.155639 25702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:15.170145 25702 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:15.172049 25702 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:15.181792 25702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6226c4cf0dfd4c31b88444f0365b8b6d: Generated new TSK 0
I20260929 02:54:15.182651 25702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:15.190207 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25610
I20260929 02:54:15.191248 25672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:15.281781 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 25610
2026-09-29T02:54:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:15 dist-test-slave-t6c1 krb5kdc[25709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:15 dist-test-slave-t6c1 krb5kdc[25709](info): set up 2 sockets
Sep 29 02:54:15 dist-test-slave-t6c1 krb5kdc[25709](info): commencing operation
krb5kdc: starting...
W20260929 02:54:16.299151 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:16 dist-test-slave-t6c1 krb5kdc[25709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:16.535209 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:16.853752 25722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:16.854058 25722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:16.854184 25722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:16.861325 25722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:16.861441 25722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:16.861514 25722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:16.861600 25722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:16.861660 25722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:16.872057 25722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:16.873629 25722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:16.875185 25722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:16.887696 25727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:16.887776 25730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:16.887634 25728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:16.888648 25722 server_base.cc:1034] running on GCE node
I20260929 02:54:16.889452 25722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:16.890878 25722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:16.892109 25722 hybrid_clock.cc:648] HybridClock initialized: now 1790650456892052 us; error 51 us; skew 500 ppm
I20260929 02:54:16.892432 25722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:16 dist-test-slave-t6c1 krb5kdc[25709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:16.895371 25722 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:16.896822 25722 webserver.cc:460] Webserver started at http://127.0.0.1:35015/ using document root <none> and password file <none>
I20260929 02:54:16.897483 25722 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:16.897599 25722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:16.897876 25722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:16.900822 25722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "313a26677e2f4dd8889497cda3d9d32e"
format_stamp: "Formatted at 2026-09-29 02:54:16 on dist-test-slave-t6c1"
server_key: "74733de3d1882ae2750caff0667a3bac"
server_key_iv: "0e26e7922a9e195fe28cb5b15ed37798"
server_key_version: "encryptionkey@0"
I20260929 02:54:16.901538 25722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "313a26677e2f4dd8889497cda3d9d32e"
format_stamp: "Formatted at 2026-09-29 02:54:16 on dist-test-slave-t6c1"
server_key: "74733de3d1882ae2750caff0667a3bac"
server_key_iv: "0e26e7922a9e195fe28cb5b15ed37798"
server_key_version: "encryptionkey@0"
I20260929 02:54:16.906483 25722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:54:16.909708 25737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:16.910956 25722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:54:16.911108 25722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "313a26677e2f4dd8889497cda3d9d32e"
format_stamp: "Formatted at 2026-09-29 02:54:16 on dist-test-slave-t6c1"
server_key: "74733de3d1882ae2750caff0667a3bac"
server_key_iv: "0e26e7922a9e195fe28cb5b15ed37798"
server_key_version: "encryptionkey@0"
I20260929 02:54:16.911276 25722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:16.938509 25722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:16.942813 25722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:16.943176 25722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:16.960007 25722 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36507
I20260929 02:54:16.960031 25788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36507 every 8 connection(s)
I20260929 02:54:16.961371 25722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:16.964990 25789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:16.966954 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 25722
I20260929 02:54:16.967115 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:16.967526 4886 external_mini_cluster.cc:1428] Setting key 5e5917c9fba200c85f2685da4c501186
I20260929 02:54:16.977628 25789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e: Bootstrap starting.
I20260929 02:54:16.981398 25789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:16.982036 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:16.969285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47503 (local address 127.0.0.1:36507)
0929 02:54:16.970075 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:16.970090 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:16.970142 (+ 52us) server_negotiation.cc:407] Connection header received
0929 02:54:16.972401 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:16.972427 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:16.972916 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:16.973328 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:16.973839 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:16.975820 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:16.977037 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:16.978017 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:16.978196 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:16.979294 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:16.979305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:16.979323 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:16.980049 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:16.980145 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:16.980176 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:16.980294 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:16.980418 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 02:54:16.980690 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":297,"threads_started":1}
I20260929 02:54:16.982671 25789 log.cc:824] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:16.985343 25789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e: No bootstrap required, opened a new log
I20260929 02:54:16.985509 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:16.994058 25789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a26677e2f4dd8889497cda3d9d32e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260929 02:54:16.994524 25789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:16.994629 25789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313a26677e2f4dd8889497cda3d9d32e, State: Initialized, Role: FOLLOWER
I20260929 02:54:16.995282 25789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a26677e2f4dd8889497cda3d9d32e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260929 02:54:16.995549 25789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:16.995707 25789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:16.995873 25789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:16.996155 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:16.989624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57639 (local address 127.0.0.1:36507)
0929 02:54:16.989840 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:16.989846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:16.990188 (+ 342us) server_negotiation.cc:407] Connection header received
0929 02:54:16.990377 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:16.990381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:16.990450 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:16.990533 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:16.991140 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:16.991895 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:16.993731 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:16.994175 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:16.994264 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:16.995445 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:16.995450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:16.995453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:16.995472 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:16.995516 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:16.995539 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:16.995625 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:16.995769 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 02:54:16.995865 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:54:16.998369 25789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a26677e2f4dd8889497cda3d9d32e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260929 02:54:16.998816 25789 leader_election.cc:302] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 313a26677e2f4dd8889497cda3d9d32e; no voters:
I20260929 02:54:16.999342 25789 leader_election.cc:288] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:16.999455 25799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:17.000115 25799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [term 1 LEADER]: Becoming Leader. State: Replica: 313a26677e2f4dd8889497cda3d9d32e, State: Running, Role: LEADER
I20260929 02:54:17.000670 25799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a26677e2f4dd8889497cda3d9d32e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260929 02:54:17.001549 25789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:17.003170 25800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "313a26677e2f4dd8889497cda3d9d32e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a26677e2f4dd8889497cda3d9d32e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } } }
I20260929 02:54:17.003598 25800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:17.004419 25808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:17.006253 25801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 313a26677e2f4dd8889497cda3d9d32e. Latest consensus state: current_term: 1 leader_uuid: "313a26677e2f4dd8889497cda3d9d32e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313a26677e2f4dd8889497cda3d9d32e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } } }
I20260929 02:54:17.006610 25801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:17.011170 25808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:17.019333 25808 catalog_manager.cc:1347] Generated new cluster ID: c8139389673547f980397d7449fc347b
I20260929 02:54:17.019434 25808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:17.035594 25808 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:17.036825 25808 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:17.047685 25808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 313a26677e2f4dd8889497cda3d9d32e: Generated new TSK 0
I20260929 02:54:17.048568 25808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:17.055446 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25722
W20260929 02:54:17.056677 25784 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47503
W20260929 02:54:17.057475 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:17.057626 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 25722
2026-09-29T02:54:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1774 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:17 dist-test-slave-t6c1 krb5kdc[25820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:17 dist-test-slave-t6c1 krb5kdc[25820](info): set up 2 sockets
Sep 29 02:54:17 dist-test-slave-t6c1 krb5kdc[25820](info): commencing operation
krb5kdc: starting...
W20260929 02:54:18.083210 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.010s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:18 dist-test-slave-t6c1 krb5kdc[25820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:18.317867 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:18.641237 25833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:18.641567 25833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:18.641703 25833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:18.649101 25833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:18.649231 25833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:18.649307 25833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:18.649389 25833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:18.649456 25833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:18.659657 25833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:18.661151 25833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:18.662586 25833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:18.674193 25839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:18.674127 25838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:18.674160 25841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:18.674983 25833 server_base.cc:1034] running on GCE node
I20260929 02:54:18.675755 25833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:18.677112 25833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:18.678316 25833 hybrid_clock.cc:648] HybridClock initialized: now 1790650458678263 us; error 39 us; skew 500 ppm
I20260929 02:54:18.678679 25833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:18 dist-test-slave-t6c1 krb5kdc[25820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:18.685468 25833 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:18.687109 25833 webserver.cc:460] Webserver started at http://127.0.0.1:35211/ using document root <none> and password file <none>
I20260929 02:54:18.687832 25833 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:18.687949 25833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:18.688266 25833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:18.691397 25833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "24f8a4d7e0b34cf59064fa690bd13aa8"
format_stamp: "Formatted at 2026-09-29 02:54:18 on dist-test-slave-t6c1"
server_key: "7e678d1e0f096eb003516768c6fed951"
server_key_iv: "ba953579f5de49d787069aa2cf9c8557"
server_key_version: "encryptionkey@0"
I20260929 02:54:18.692174 25833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "24f8a4d7e0b34cf59064fa690bd13aa8"
format_stamp: "Formatted at 2026-09-29 02:54:18 on dist-test-slave-t6c1"
server_key: "7e678d1e0f096eb003516768c6fed951"
server_key_iv: "ba953579f5de49d787069aa2cf9c8557"
server_key_version: "encryptionkey@0"
I20260929 02:54:18.697369 25833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:54:18.700769 25848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:18.702036 25833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:54:18.702198 25833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "24f8a4d7e0b34cf59064fa690bd13aa8"
format_stamp: "Formatted at 2026-09-29 02:54:18 on dist-test-slave-t6c1"
server_key: "7e678d1e0f096eb003516768c6fed951"
server_key_iv: "ba953579f5de49d787069aa2cf9c8557"
server_key_version: "encryptionkey@0"
I20260929 02:54:18.702381 25833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:18.707911 25833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:18.711925 25833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:18.712172 25833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:18.728324 25833 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43409
I20260929 02:54:18.728366 25899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43409 every 8 connection(s)
I20260929 02:54:18.729733 25833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:18.730185 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 25833
I20260929 02:54:18.730348 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:18.730909 4886 external_mini_cluster.cc:1428] Setting key 544da7342523449a297b4d42ecd4f37b
I20260929 02:54:18.734201 25900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:18.739490 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:18.732663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52527 (local address 127.0.0.1:43409)
0929 02:54:18.733451 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:18.733465 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:18.733515 (+ 50us) server_negotiation.cc:407] Connection header received
0929 02:54:18.735722 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:18.735749 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:18.736164 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:18.736594 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:18.737544 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:18.737556 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:18.737572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:18.737588 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:18.737675 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:18.737700 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:18.737918 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:18.738045 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 02:54:18.738228 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":305,"threads_started":1}
I20260929 02:54:18.743418 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:18.746806 25900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8: Bootstrap starting.
I20260929 02:54:18.749770 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:18.747526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37099 (local address 127.0.0.1:43409)
0929 02:54:18.747701 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:18.747707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:18.748043 (+ 336us) server_negotiation.cc:407] Connection header received
0929 02:54:18.748205 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:18.748210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:18.748260 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:18.748350 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:18.749184 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:18.749190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:18.749195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:18.749213 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:18.749249 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:18.749263 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:18.749361 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:18.749451 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 02:54:18.749508 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:54:18.750125 25900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:18.751241 25900 log.cc:824] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:18.753767 25900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8: No bootstrap required, opened a new log
I20260929 02:54:18.760846 25900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8a4d7e0b34cf59064fa690bd13aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } }
I20260929 02:54:18.761214 25900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:18.761309 25900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f8a4d7e0b34cf59064fa690bd13aa8, State: Initialized, Role: FOLLOWER
I20260929 02:54:18.761922 25900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8a4d7e0b34cf59064fa690bd13aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } }
I20260929 02:54:18.762146 25900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:18.762279 25900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:18.762435 25900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:18.764732 25900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8a4d7e0b34cf59064fa690bd13aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } }
I20260929 02:54:18.765266 25900 leader_election.cc:302] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24f8a4d7e0b34cf59064fa690bd13aa8; no voters:
I20260929 02:54:18.765811 25900 leader_election.cc:288] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:18.765946 25910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:18.766599 25910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [term 1 LEADER]: Becoming Leader. State: Replica: 24f8a4d7e0b34cf59064fa690bd13aa8, State: Running, Role: LEADER
I20260929 02:54:18.767184 25910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8a4d7e0b34cf59064fa690bd13aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } }
I20260929 02:54:18.767922 25900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:18.769944 25911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24f8a4d7e0b34cf59064fa690bd13aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8a4d7e0b34cf59064fa690bd13aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } } }
I20260929 02:54:18.770318 25911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:18.770522 25912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24f8a4d7e0b34cf59064fa690bd13aa8. Latest consensus state: current_term: 1 leader_uuid: "24f8a4d7e0b34cf59064fa690bd13aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8a4d7e0b34cf59064fa690bd13aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43409 } } }
I20260929 02:54:18.771612 25912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:18.772146 25917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:18.777576 25917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:18.788055 25917 catalog_manager.cc:1347] Generated new cluster ID: c8d3eea1c7d5494ca40beb251ac03c01
I20260929 02:54:18.788178 25917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:18.799261 25917 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:18.799672 25917 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:18.809051 25917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24f8a4d7e0b34cf59064fa690bd13aa8: Generated new TSK 0
I20260929 02:54:18.809983 25917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:18.819934 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25833
W20260929 02:54:18.820947 25895 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52527
W20260929 02:54:18.821552 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25833: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:18.821699 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 25833
2026-09-29T02:54:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1766 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:18 dist-test-slave-t6c1 krb5kdc[25931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:18 dist-test-slave-t6c1 krb5kdc[25931](info): set up 2 sockets
Sep 29 02:54:18 dist-test-slave-t6c1 krb5kdc[25931](info): commencing operation
krb5kdc: starting...
W20260929 02:54:19.835130 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:19 dist-test-slave-t6c1 krb5kdc[25931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:20.070096 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:20.394369 25944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:20.394665 25944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:20.394764 25944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:20.401892 25944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:20.401999 25944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:20.402052 25944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:20.402117 25944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:20.402160 25944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:20.411768 25944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:20.412907 25944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:20.414115 25944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:20.424916 25952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:20.425020 25949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:20.424973 25950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:20.426015 25944 server_base.cc:1034] running on GCE node
I20260929 02:54:20.426795 25944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:20.428229 25944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:20.429422 25944 hybrid_clock.cc:648] HybridClock initialized: now 1790650460429375 us; error 38 us; skew 500 ppm
I20260929 02:54:20.429756 25944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:20 dist-test-slave-t6c1 krb5kdc[25931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:20.432768 25944 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:20.434253 25944 webserver.cc:460] Webserver started at http://127.0.0.1:41879/ using document root <none> and password file <none>
I20260929 02:54:20.434973 25944 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:20.435091 25944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:20.435380 25944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:20.438370 25944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "bb5ffa675cc14446bc285922c0c1cd6b"
format_stamp: "Formatted at 2026-09-29 02:54:20 on dist-test-slave-t6c1"
server_key: "e815333e7d73f33d35b4848dde8047c2"
server_key_iv: "fa336648b1377d4932b3969a84dc4680"
server_key_version: "encryptionkey@0"
I20260929 02:54:20.439183 25944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "bb5ffa675cc14446bc285922c0c1cd6b"
format_stamp: "Formatted at 2026-09-29 02:54:20 on dist-test-slave-t6c1"
server_key: "e815333e7d73f33d35b4848dde8047c2"
server_key_iv: "fa336648b1377d4932b3969a84dc4680"
server_key_version: "encryptionkey@0"
I20260929 02:54:20.444314 25944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:54:20.447798 25959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:20.449440 25944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:54:20.449643 25944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "bb5ffa675cc14446bc285922c0c1cd6b"
format_stamp: "Formatted at 2026-09-29 02:54:20 on dist-test-slave-t6c1"
server_key: "e815333e7d73f33d35b4848dde8047c2"
server_key_iv: "fa336648b1377d4932b3969a84dc4680"
server_key_version: "encryptionkey@0"
I20260929 02:54:20.449842 25944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:20.477840 25944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:20.481860 25944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:20.482141 25944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:20.499033 25944 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44247
I20260929 02:54:20.499078 26010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44247 every 8 connection(s)
I20260929 02:54:20.500340 25944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:20.502328 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 25944
I20260929 02:54:20.502528 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:20.503122 4886 external_mini_cluster.cc:1428] Setting key c23f19145759d9171f9eaea7f4aa6de8
I20260929 02:54:20.504403 26011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:20.516160 26011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b: Bootstrap starting.
Sep 29 02:54:20 dist-test-slave-t6c1 krb5kdc[25931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:54:20.519554 26011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:20.520584 26011 log.cc:824] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:20.523445 26011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b: No bootstrap required, opened a new log
I20260929 02:54:20.524940 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:20.504870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:49983 (local address 127.0.0.1:44247)
0929 02:54:20.505575 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:20.505587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:20.505625 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:54:20.507194 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:20.507215 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:20.507599 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:20.507978 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:20.509375 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:20.510666 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:20.512111 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:20.513176 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:20.513358 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:20.518129 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:20.518137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:20.518149 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:20.518188 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:20.521060 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:20.521750 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:20.521756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:20.521761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:20.521922 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:20.522411 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:20.522414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:20.522417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:20.522909 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:20.523228 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:20.523677 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 02:54:20.523991 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":295,"threads_started":1}
I20260929 02:54:20.528580 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:20.532171 26011 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5ffa675cc14446bc285922c0c1cd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44247 } }
I20260929 02:54:20.532622 26011 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:20.532724 26011 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5ffa675cc14446bc285922c0c1cd6b, State: Initialized, Role: FOLLOWER
I20260929 02:54:20.533324 26011 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5ffa675cc14446bc285922c0c1cd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44247 } }
I20260929 02:54:20.533602 26011 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:20.533751 26011 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:20.533960 26011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:20.536489 26011 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5ffa675cc14446bc285922c0c1cd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44247 } }
I20260929 02:54:20.537097 26011 leader_election.cc:302] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb5ffa675cc14446bc285922c0c1cd6b; no voters:
I20260929 02:54:20.537714 26011 leader_election.cc:288] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:20.537811 26021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:20.538848 26021 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [term 1 LEADER]: Becoming Leader. State: Replica: bb5ffa675cc14446bc285922c0c1cd6b, State: Running, Role: LEADER
I20260929 02:54:20.539506 26021 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5ffa675cc14446bc285922c0c1cd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44247 } }
I20260929 02:54:20.540088 26011 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:20.543087 26022 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb5ffa675cc14446bc285922c0c1cd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5ffa675cc14446bc285922c0c1cd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44247 } } }
I20260929 02:54:20.543887 26022 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:20.544636 26023 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb5ffa675cc14446bc285922c0c1cd6b. Latest consensus state: current_term: 1 leader_uuid: "bb5ffa675cc14446bc285922c0c1cd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5ffa675cc14446bc285922c0c1cd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44247 } } }
I20260929 02:54:20.544932 26030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:20.545266 26023 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:20.550773 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:20.533750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:41431 (local address 127.0.0.1:44247)
0929 02:54:20.533917 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:20.533925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:20.534281 (+ 356us) server_negotiation.cc:407] Connection header received
0929 02:54:20.534405 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:20.534410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:20.534487 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:20.534593 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:20.535883 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:20.536942 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:20.538336 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:20.539067 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:20.539193 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:20.542491 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:20.542500 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:20.542506 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:20.542550 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:20.546828 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:20.547964 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:20.547971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:20.547975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:20.548153 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:20.548723 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:20.548729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:20.548733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:20.549155 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:20.549430 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:20.550165 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 02:54:20.550351 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:54:20.553691 26030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:20.562714 26030 catalog_manager.cc:1347] Generated new cluster ID: 0eab73855e214b348f9a09ab2fcffbc8
I20260929 02:54:20.562847 26030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:20.578538 26030 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:20.579921 26030 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:20.585942 26030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb5ffa675cc14446bc285922c0c1cd6b: Generated new TSK 0
I20260929 02:54:20.587062 26030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:20.600858 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 25944
I20260929 02:54:20.601998 26006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:20.691825 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 25944
W20260929 02:54:20.705636 25929 connection.cc:535] client connection to 127.0.0.1:44247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44247): Connection reset by peer (error 104)
2026-09-29T02:54:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:20 dist-test-slave-t6c1 krb5kdc[26043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:20 dist-test-slave-t6c1 krb5kdc[26043](info): set up 2 sockets
Sep 29 02:54:20 dist-test-slave-t6c1 krb5kdc[26043](info): commencing operation
krb5kdc: starting...
W20260929 02:54:21.715094 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:21 dist-test-slave-t6c1 krb5kdc[26043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:21.954200 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:22.269951 26056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:22.270284 26056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:22.270409 26056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:22.277709 26056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:22.277853 26056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:22.277930 26056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:22.278007 26056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:22.278066 26056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:22.288051 26056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:22.289400 26056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:22.290671 26056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:22.301497 26061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:22.301437 26064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:22.301469 26062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:22.302582 26056 server_base.cc:1034] running on GCE node
I20260929 02:54:22.303282 26056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:22.304493 26056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:22.305675 26056 hybrid_clock.cc:648] HybridClock initialized: now 1790650462305639 us; error 37 us; skew 500 ppm
I20260929 02:54:22.305960 26056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:22 dist-test-slave-t6c1 krb5kdc[26043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:22.308712 26056 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:22.310041 26056 webserver.cc:460] Webserver started at http://127.0.0.1:40699/ using document root <none> and password file <none>
I20260929 02:54:22.310652 26056 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:22.310762 26056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:22.311075 26056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:22.313774 26056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "8586eb21478f461a9611cb1c0e68bf6b"
format_stamp: "Formatted at 2026-09-29 02:54:22 on dist-test-slave-t6c1"
server_key: "fe1894860674c7dab33eb482866ab4f0"
server_key_iv: "681956a59e39c2d9a8480624a95d9735"
server_key_version: "encryptionkey@0"
I20260929 02:54:22.314479 26056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "8586eb21478f461a9611cb1c0e68bf6b"
format_stamp: "Formatted at 2026-09-29 02:54:22 on dist-test-slave-t6c1"
server_key: "fe1894860674c7dab33eb482866ab4f0"
server_key_iv: "681956a59e39c2d9a8480624a95d9735"
server_key_version: "encryptionkey@0"
I20260929 02:54:22.319041 26056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260929 02:54:22.322021 26071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:22.323233 26056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 02:54:22.323378 26056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "8586eb21478f461a9611cb1c0e68bf6b"
format_stamp: "Formatted at 2026-09-29 02:54:22 on dist-test-slave-t6c1"
server_key: "fe1894860674c7dab33eb482866ab4f0"
server_key_iv: "681956a59e39c2d9a8480624a95d9735"
server_key_version: "encryptionkey@0"
I20260929 02:54:22.323544 26056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:22.347810 26056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:22.351775 26056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:22.352052 26056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:22.369410 26056 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32787
I20260929 02:54:22.369432 26122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32787 every 8 connection(s)
I20260929 02:54:22.370785 26056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:22.374240 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 26056
I20260929 02:54:22.374402 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:22.374541 26123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:22.374856 4886 external_mini_cluster.cc:1428] Setting key d432beac2c5eedf099149ea8ac409eda
I20260929 02:54:22.387825 26126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:22.376602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:53963 (local address 127.0.0.1:32787)
0929 02:54:22.377386 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:22.377397 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:22.377440 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:54:22.379109 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:22.379130 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:22.379538 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:22.379885 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:22.380460 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:22.381772 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:22.383533 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:22.384201 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:22.384330 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:22.385571 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:22.385579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:22.385592 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:22.386155 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:22.386236 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:22.386268 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:22.386382 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:22.386633 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 02:54:22.386803 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":343,"threads_started":1}
I20260929 02:54:22.388063 26123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b: Bootstrap starting.
I20260929 02:54:22.391307 26123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:22.391449 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:22.392503 26123 log.cc:824] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:22.395650 26123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b: No bootstrap required, opened a new log
I20260929 02:54:22.401865 26126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:22.396198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:56023 (local address 127.0.0.1:32787)
0929 02:54:22.396347 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:22.396354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:22.396744 (+ 390us) server_negotiation.cc:407] Connection header received
0929 02:54:22.396908 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:22.396912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:22.396971 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:22.397048 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:22.397544 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:22.398310 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:22.399633 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:22.400074 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:22.400184 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:22.401196 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:22.401200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:22.401204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:22.401220 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:22.401256 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:22.401270 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:22.401375 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:22.401547 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 02:54:22.401607 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 02:54:22.403384 26123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586eb21478f461a9611cb1c0e68bf6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } }
I20260929 02:54:22.403717 26123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:22.403805 26123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586eb21478f461a9611cb1c0e68bf6b, State: Initialized, Role: FOLLOWER
I20260929 02:54:22.404306 26123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586eb21478f461a9611cb1c0e68bf6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } }
I20260929 02:54:22.404515 26123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:22.404635 26123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:22.404770 26123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:22.406823 26123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586eb21478f461a9611cb1c0e68bf6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } }
I20260929 02:54:22.407387 26123 leader_election.cc:302] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8586eb21478f461a9611cb1c0e68bf6b; no voters:
I20260929 02:54:22.407876 26123 leader_election.cc:288] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:22.407970 26133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:22.408567 26133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [term 1 LEADER]: Becoming Leader. State: Replica: 8586eb21478f461a9611cb1c0e68bf6b, State: Running, Role: LEADER
I20260929 02:54:22.409029 26133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586eb21478f461a9611cb1c0e68bf6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } }
I20260929 02:54:22.409663 26123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:22.411424 26135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8586eb21478f461a9611cb1c0e68bf6b. Latest consensus state: current_term: 1 leader_uuid: "8586eb21478f461a9611cb1c0e68bf6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586eb21478f461a9611cb1c0e68bf6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } } }
I20260929 02:54:22.411805 26135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:22.412480 26134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8586eb21478f461a9611cb1c0e68bf6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8586eb21478f461a9611cb1c0e68bf6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32787 } } }
I20260929 02:54:22.412828 26134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:22.413149 26139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:22.417626 26139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:22.425468 26139 catalog_manager.cc:1347] Generated new cluster ID: 08cb7465a5c9415eb1b8d50b132fb124
I20260929 02:54:22.425587 26139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:22.438651 26139 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:22.440012 26139 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:22.446544 26139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8586eb21478f461a9611cb1c0e68bf6b: Generated new TSK 0
I20260929 02:54:22.447418 26139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:22.453070 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26056
W20260929 02:54:22.454176 26118 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.73:53963
W20260929 02:54:22.454887 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26056: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:22.455063 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 26056
W20260929 02:54:22.468446 26041 connection.cc:535] client connection to 127.0.0.1:32787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32787): Connection reset by peer (error 104)
2026-09-29T02:54:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1762 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:22 dist-test-slave-t6c1 krb5kdc[26154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:22 dist-test-slave-t6c1 krb5kdc[26154](info): set up 2 sockets
Sep 29 02:54:22 dist-test-slave-t6c1 krb5kdc[26154](info): commencing operation
krb5kdc: starting...
W20260929 02:54:23.462989 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:23 dist-test-slave-t6c1 krb5kdc[26154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:23.699957 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:24.021303 26167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:24.021581 26167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:24.021708 26167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:24.028916 26167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:24.029044 26167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:24.029119 26167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:24.029198 26167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:24.029256 26167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:24.039402 26167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:24.040688 26167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:24.041924 26167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:24.052857 26172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:24.052891 26173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:24.052819 26175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:24.053596 26167 server_base.cc:1034] running on GCE node
I20260929 02:54:24.054294 26167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:24.055792 26167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:24.057008 26167 hybrid_clock.cc:648] HybridClock initialized: now 1790650464056964 us; error 41 us; skew 500 ppm
I20260929 02:54:24.057349 26167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:24 dist-test-slave-t6c1 krb5kdc[26154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:24.064517 26167 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:24.066147 26167 webserver.cc:460] Webserver started at http://127.0.0.1:45925/ using document root <none> and password file <none>
I20260929 02:54:24.066862 26167 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:24.067018 26167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:24.067310 26167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:24.070266 26167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "b7fb47fa24284f2e9e855957b8baf7c6"
format_stamp: "Formatted at 2026-09-29 02:54:24 on dist-test-slave-t6c1"
server_key: "51f2280e382b824848edfd820efd7262"
server_key_iv: "3958563449b5998341d24c9cb82ce82d"
server_key_version: "encryptionkey@0"
I20260929 02:54:24.071039 26167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "b7fb47fa24284f2e9e855957b8baf7c6"
format_stamp: "Formatted at 2026-09-29 02:54:24 on dist-test-slave-t6c1"
server_key: "51f2280e382b824848edfd820efd7262"
server_key_iv: "3958563449b5998341d24c9cb82ce82d"
server_key_version: "encryptionkey@0"
I20260929 02:54:24.076061 26167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:54:24.079339 26182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:24.080616 26167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:54:24.080767 26167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "b7fb47fa24284f2e9e855957b8baf7c6"
format_stamp: "Formatted at 2026-09-29 02:54:24 on dist-test-slave-t6c1"
server_key: "51f2280e382b824848edfd820efd7262"
server_key_iv: "3958563449b5998341d24c9cb82ce82d"
server_key_version: "encryptionkey@0"
I20260929 02:54:24.080931 26167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:24.086344 26167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:24.089959 26167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:24.090212 26167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:24.106566 26167 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45927
I20260929 02:54:24.106598 26233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45927 every 8 connection(s)
I20260929 02:54:24.108024 26167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:24.109901 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 26167
I20260929 02:54:24.110100 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:24.110625 4886 external_mini_cluster.cc:1428] Setting key 7bd802241201a86262c7d7a824d75848
I20260929 02:54:24.112489 26234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:24.119014 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:24.112359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:55515 (local address 127.0.0.1:45927)
0929 02:54:24.113130 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:24.113145 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:24.113193 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:54:24.115311 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:24.115339 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:24.115791 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:24.116200 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:24.117142 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:24.117149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:24.117166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:24.117183 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:24.117275 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:24.117301 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:24.117501 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:24.117629 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 02:54:24.117835 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":343,"threads_started":1}
I20260929 02:54:24.122184 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:24.125897 26234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6: Bootstrap starting.
I20260929 02:54:24.129462 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:24.127328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:60427 (local address 127.0.0.1:45927)
0929 02:54:24.127504 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:24.127510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:24.127811 (+ 301us) server_negotiation.cc:407] Connection header received
0929 02:54:24.127967 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:24.127970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:24.128017 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:24.128084 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:24.128940 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:24.128944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:24.128947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:24.128960 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:24.128993 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:24.129007 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:24.129083 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:24.129188 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 02:54:24.129228 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:54:24.129582 26234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:24.130723 26234 log.cc:824] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:24.133173 26234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6: No bootstrap required, opened a new log
I20260929 02:54:24.140024 26234 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb47fa24284f2e9e855957b8baf7c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260929 02:54:24.140333 26234 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:24.140403 26234 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fb47fa24284f2e9e855957b8baf7c6, State: Initialized, Role: FOLLOWER
I20260929 02:54:24.140853 26234 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb47fa24284f2e9e855957b8baf7c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260929 02:54:24.141039 26234 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:24.141177 26234 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:24.141330 26234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:24.143419 26234 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb47fa24284f2e9e855957b8baf7c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260929 02:54:24.143837 26234 leader_election.cc:302] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7fb47fa24284f2e9e855957b8baf7c6; no voters:
I20260929 02:54:24.144306 26234 leader_election.cc:288] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:24.144405 26244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:24.144982 26244 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [term 1 LEADER]: Becoming Leader. State: Replica: b7fb47fa24284f2e9e855957b8baf7c6, State: Running, Role: LEADER
I20260929 02:54:24.145440 26244 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb47fa24284f2e9e855957b8baf7c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260929 02:54:24.146100 26234 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:24.147998 26245 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7fb47fa24284f2e9e855957b8baf7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb47fa24284f2e9e855957b8baf7c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } } }
I20260929 02:54:24.148451 26245 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:24.148017 26246 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7fb47fa24284f2e9e855957b8baf7c6. Latest consensus state: current_term: 1 leader_uuid: "b7fb47fa24284f2e9e855957b8baf7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb47fa24284f2e9e855957b8baf7c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } } }
I20260929 02:54:24.149102 26246 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:24.149919 26250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:24.158707 26250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:24.168030 26250 catalog_manager.cc:1347] Generated new cluster ID: ee472807ca394773b8bbc0b1fc1168ed
I20260929 02:54:24.168157 26250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:24.194069 26250 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:24.194574 26250 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:24.206625 26250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7fb47fa24284f2e9e855957b8baf7c6: Generated new TSK 0
I20260929 02:54:24.207520 26250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:24.221958 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26167
W20260929 02:54:24.223055 26229 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.73:55515
W20260929 02:54:24.223722 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26167: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:24.223879 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 26167
2026-09-29T02:54:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1769 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:24 dist-test-slave-t6c1 krb5kdc[26265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:24 dist-test-slave-t6c1 krb5kdc[26265](info): set up 2 sockets
Sep 29 02:54:24 dist-test-slave-t6c1 krb5kdc[26265](info): commencing operation
krb5kdc: starting...
W20260929 02:54:25.231207 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:25 dist-test-slave-t6c1 krb5kdc[26265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:25.468525 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:25.785943 26278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:25.786237 26278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:25.786375 26278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:25.793411 26278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:25.793519 26278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:25.793601 26278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:25.793680 26278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:25.793738 26278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:25.803154 26278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:25.804333 26278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:25.805547 26278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:25.816182 26283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:25.816277 26284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:25.816704 26286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:25.817044 26278 server_base.cc:1034] running on GCE node
I20260929 02:54:25.817701 26278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:25.818902 26278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:25.820122 26278 hybrid_clock.cc:648] HybridClock initialized: now 1790650465820071 us; error 46 us; skew 500 ppm
I20260929 02:54:25.820426 26278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:25 dist-test-slave-t6c1 krb5kdc[26265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:25.823227 26278 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:25.824549 26278 webserver.cc:460] Webserver started at http://127.0.0.1:36855/ using document root <none> and password file <none>
I20260929 02:54:25.825163 26278 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:25.825270 26278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:25.825522 26278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:25.828263 26278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "2646e16a9d13455a8970bacaadd39134"
format_stamp: "Formatted at 2026-09-29 02:54:25 on dist-test-slave-t6c1"
server_key: "823c9035cd1b20c804d5be6a7adf5b8a"
server_key_iv: "c74ef0728d9d5f5e779cb9a83b34f7cf"
server_key_version: "encryptionkey@0"
I20260929 02:54:25.828941 26278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "2646e16a9d13455a8970bacaadd39134"
format_stamp: "Formatted at 2026-09-29 02:54:25 on dist-test-slave-t6c1"
server_key: "823c9035cd1b20c804d5be6a7adf5b8a"
server_key_iv: "c74ef0728d9d5f5e779cb9a83b34f7cf"
server_key_version: "encryptionkey@0"
I20260929 02:54:25.833500 26278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:54:25.836556 26293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:25.837702 26278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:54:25.837845 26278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "2646e16a9d13455a8970bacaadd39134"
format_stamp: "Formatted at 2026-09-29 02:54:25 on dist-test-slave-t6c1"
server_key: "823c9035cd1b20c804d5be6a7adf5b8a"
server_key_iv: "c74ef0728d9d5f5e779cb9a83b34f7cf"
server_key_version: "encryptionkey@0"
I20260929 02:54:25.838001 26278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:25.853803 26278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:25.857234 26278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:25.857462 26278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:25.873528 26278 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40855
I20260929 02:54:25.873549 26344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40855 every 8 connection(s)
I20260929 02:54:25.874783 26278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:25.878324 26345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:25.880259 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 26278
I20260929 02:54:25.880445 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:25.880967 4886 external_mini_cluster.cc:1428] Setting key a816ba1fe7310ae22eff944050f571a0
I20260929 02:54:25.891520 26345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134: Bootstrap starting.
Sep 29 02:54:25 dist-test-slave-t6c1 krb5kdc[26265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:54:25.895031 26345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:25.896109 26345 log.cc:824] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:25.899261 26345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134: No bootstrap required, opened a new log
I20260929 02:54:25.903335 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:25.882609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:52913 (local address 127.0.0.1:40855)
0929 02:54:25.883369 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:25.883383 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:25.883421 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:54:25.884939 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:25.884960 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:25.885330 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:25.885608 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:25.887305 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:25.888552 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:25.890303 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:25.890907 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:25.891068 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:25.895583 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:25.895590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:25.895602 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:25.895635 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:25.899123 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:25.899828 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:25.899836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:25.899843 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:25.900049 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:25.900418 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:25.900422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:25.900425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:25.901086 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:25.901388 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:25.901705 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 02:54:25.902034 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":326,"threads_started":1}
I20260929 02:54:25.906951 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:25.908396 26345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2646e16a9d13455a8970bacaadd39134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260929 02:54:25.908722 26345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:25.908811 26345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2646e16a9d13455a8970bacaadd39134, State: Initialized, Role: FOLLOWER
I20260929 02:54:25.909389 26345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2646e16a9d13455a8970bacaadd39134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260929 02:54:25.909618 26345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:25.909742 26345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:25.909886 26345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:25.912317 26345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2646e16a9d13455a8970bacaadd39134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260929 02:54:25.912756 26345 leader_election.cc:302] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2646e16a9d13455a8970bacaadd39134; no voters:
I20260929 02:54:25.913313 26345 leader_election.cc:288] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:25.913439 26355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:25.915512 26345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:25.920305 26355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [term 1 LEADER]: Becoming Leader. State: Replica: 2646e16a9d13455a8970bacaadd39134, State: Running, Role: LEADER
I20260929 02:54:25.920921 26355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2646e16a9d13455a8970bacaadd39134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260929 02:54:25.923521 26356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2646e16a9d13455a8970bacaadd39134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2646e16a9d13455a8970bacaadd39134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } } }
I20260929 02:54:25.923852 26356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:25.923939 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:25.910948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:36051 (local address 127.0.0.1:40855)
0929 02:54:25.911091 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:25.911096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:25.911497 (+ 401us) server_negotiation.cc:407] Connection header received
0929 02:54:25.911650 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:25.911656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:25.911716 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:25.911791 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:25.913132 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:25.914232 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:25.915620 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:25.916119 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:25.916221 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:25.918396 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:25.918402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:25.918406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:25.918442 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:25.921083 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:25.921757 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:25.921762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:25.921765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:25.921889 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:25.922267 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:25.922272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:25.922276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:25.922683 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:25.922985 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:25.923429 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 02:54:25.923597 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 02:54:25.924345 26357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2646e16a9d13455a8970bacaadd39134. Latest consensus state: current_term: 1 leader_uuid: "2646e16a9d13455a8970bacaadd39134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2646e16a9d13455a8970bacaadd39134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } } }
I20260929 02:54:25.924640 26357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:25.925401 26362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:25.930249 26362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:25.945284 26362 catalog_manager.cc:1347] Generated new cluster ID: 2b21264ff8f9413ca15865246f111ccd
I20260929 02:54:25.945377 26362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:25.974789 26362 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:25.976605 26362 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:25.985792 26362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2646e16a9d13455a8970bacaadd39134: Generated new TSK 0
I20260929 02:54:25.986716 26362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:25.993546 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26278
I20260929 02:54:25.995299 26340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:26.085317 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 26278
W20260929 02:54:26.098681 26263 connection.cc:535] client connection to 127.0.0.1:40855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40855): Connection reset by peer (error 104)
2026-09-29T02:54:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1860 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:26 dist-test-slave-t6c1 krb5kdc[26377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:26 dist-test-slave-t6c1 krb5kdc[26377](info): set up 2 sockets
Sep 29 02:54:26 dist-test-slave-t6c1 krb5kdc[26377](info): commencing operation
krb5kdc: starting...
W20260929 02:54:27.087117 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:27 dist-test-slave-t6c1 krb5kdc[26377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:27.294137 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:27.597360 26390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:27.597616 26390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:27.597709 26390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:27.604571 26390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:27.604667 26390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:27.604718 26390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:27.604779 26390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:27.604820 26390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:27.613940 26390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44873
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:27.615044 26390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:27.616209 26390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:27.626657 26395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:27.626709 26396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:27.626833 26398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:27.627630 26390 server_base.cc:1034] running on GCE node
I20260929 02:54:27.628286 26390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:27.629482 26390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:27.630671 26390 hybrid_clock.cc:648] HybridClock initialized: now 1790650467630628 us; error 32 us; skew 500 ppm
I20260929 02:54:27.631002 26390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:27 dist-test-slave-t6c1 krb5kdc[26377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:27.633754 26390 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:27.635079 26390 webserver.cc:460] Webserver started at http://127.0.0.1:45139/ using document root <none> and password file <none>
I20260929 02:54:27.635667 26390 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:27.635773 26390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:27.636030 26390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:27.638688 26390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "cd56fe17f17a423da45ed444b0e737a2"
format_stamp: "Formatted at 2026-09-29 02:54:27 on dist-test-slave-t6c1"
server_key: "0b7399b815aac7740635967da4004018"
server_key_iv: "d8d190aeab3eef56a7825625d521c839"
server_key_version: "encryptionkey@0"
I20260929 02:54:27.639436 26390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "cd56fe17f17a423da45ed444b0e737a2"
format_stamp: "Formatted at 2026-09-29 02:54:27 on dist-test-slave-t6c1"
server_key: "0b7399b815aac7740635967da4004018"
server_key_iv: "d8d190aeab3eef56a7825625d521c839"
server_key_version: "encryptionkey@0"
I20260929 02:54:27.643975 26390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 02:54:27.647078 26405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:27.648240 26390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:54:27.648378 26390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "cd56fe17f17a423da45ed444b0e737a2"
format_stamp: "Formatted at 2026-09-29 02:54:27 on dist-test-slave-t6c1"
server_key: "0b7399b815aac7740635967da4004018"
server_key_iv: "d8d190aeab3eef56a7825625d521c839"
server_key_version: "encryptionkey@0"
I20260929 02:54:27.648535 26390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:27.662600 26390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:27.665962 26390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:27.666188 26390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:27.682134 26390 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44873
I20260929 02:54:27.682174 26456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44873 every 8 connection(s)
I20260929 02:54:27.683349 26390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:27.686620 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:27.691179 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 26390
I20260929 02:54:27.691361 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:27.691895 4886 external_mini_cluster.cc:1428] Setting key 2159b3923f80ed5e2c1fbc578e2a6a32
I20260929 02:54:27.697048 26457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2: Bootstrap starting.
I20260929 02:54:27.700301 26457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:27.701354 26457 log.cc:824] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:27.704200 26457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2: No bootstrap required, opened a new log
I20260929 02:54:27.704478 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:27.693541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:56663 (local address 127.0.0.1:44873)
0929 02:54:27.694214 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:27.694224 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:27.694261 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:54:27.695964 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:27.695983 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:27.696327 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:27.696607 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:27.697160 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:27.698399 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:27.700189 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:27.700995 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:27.701123 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:27.702352 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:27.702362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:27.702379 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:27.702943 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:27.703018 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:27.703040 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:27.703143 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:27.703427 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 02:54:27.703570 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":301,"threads_started":1}
I20260929 02:54:27.707829 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:27.713076 26457 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd56fe17f17a423da45ed444b0e737a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } }
I20260929 02:54:27.713493 26457 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:27.713610 26457 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd56fe17f17a423da45ed444b0e737a2, State: Initialized, Role: FOLLOWER
I20260929 02:54:27.714191 26457 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd56fe17f17a423da45ed444b0e737a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } }
I20260929 02:54:27.714468 26457 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:27.714609 26457 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:27.714769 26457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:27.717226 26457 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd56fe17f17a423da45ed444b0e737a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } }
I20260929 02:54:27.717744 26457 leader_election.cc:302] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd56fe17f17a423da45ed444b0e737a2; no voters:
I20260929 02:54:27.718297 26457 leader_election.cc:288] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:27.718379 26467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:27.719116 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:27.712602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:51257 (local address 127.0.0.1:44873)
0929 02:54:27.712756 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:27.712761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:27.713167 (+ 406us) server_negotiation.cc:407] Connection header received
0929 02:54:27.713297 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:27.713302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:27.713378 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:27.713477 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:27.714041 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:27.715141 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:27.716448 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:27.717095 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:27.717224 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:27.718158 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:27.718165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:27.718171 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:27.718192 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:27.718261 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:27.718288 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:27.718410 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:27.718678 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 02:54:27.718736 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:54:27.723677 26467 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [term 1 LEADER]: Becoming Leader. State: Replica: cd56fe17f17a423da45ed444b0e737a2, State: Running, Role: LEADER
I20260929 02:54:27.724139 26467 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd56fe17f17a423da45ed444b0e737a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } }
I20260929 02:54:27.724617 26457 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:27.726483 26469 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd56fe17f17a423da45ed444b0e737a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd56fe17f17a423da45ed444b0e737a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } } }
I20260929 02:54:27.726828 26469 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:27.727373 26468 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd56fe17f17a423da45ed444b0e737a2. Latest consensus state: current_term: 1 leader_uuid: "cd56fe17f17a423da45ed444b0e737a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd56fe17f17a423da45ed444b0e737a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } } }
I20260929 02:54:27.727625 26468 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:27.727736 26474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:27.734357 26474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:27.742626 26474 catalog_manager.cc:1347] Generated new cluster ID: 7595daf79ea9466db6b6b8bcb7802346
I20260929 02:54:27.742719 26474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:27.753492 26474 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:27.755168 26474 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:27.764802 26474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd56fe17f17a423da45ed444b0e737a2: Generated new TSK 0
I20260929 02:54:27.765477 26474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:27.775074 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26390
W20260929 02:54:27.776319 26452 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.73:56663
W20260929 02:54:27.776963 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26390: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:27.777074 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 26390
W20260929 02:54:27.789826 26375 connection.cc:535] client connection to 127.0.0.1:44873 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44873): Connection reset by peer (error 104)
2026-09-29T02:54:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1691 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:27 dist-test-slave-t6c1 krb5kdc[26488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:27 dist-test-slave-t6c1 krb5kdc[26488](info): set up 2 sockets
Sep 29 02:54:27 dist-test-slave-t6c1 krb5kdc[26488](info): commencing operation
krb5kdc: starting...
W20260929 02:54:28.782991 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:28 dist-test-slave-t6c1 krb5kdc[26488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:54:29.001798 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:29.317087 26501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:29.317365 26501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:29.317488 26501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:29.324666 26501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:29.324775 26501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:29.324852 26501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:29.324928 26501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:29.324986 26501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:29.335790 26501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:29.337296 26501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:29.338652 26501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:29.350448 26506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:29.350414 26509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:29.350392 26507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:29.351169 26501 server_base.cc:1034] running on GCE node
I20260929 02:54:29.351959 26501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:29.353341 26501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:29.354545 26501 hybrid_clock.cc:648] HybridClock initialized: now 1790650469354496 us; error 45 us; skew 500 ppm
I20260929 02:54:29.354889 26501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:29 dist-test-slave-t6c1 krb5kdc[26488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:29.363291 26501 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:54:29.365003 26501 webserver.cc:460] Webserver started at http://127.0.0.1:46301/ using document root <none> and password file <none>
I20260929 02:54:29.365751 26501 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:29.365885 26501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:29.366209 26501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:29.369418 26501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "1a09090f284d47db82d27a9463fad33e"
format_stamp: "Formatted at 2026-09-29 02:54:29 on dist-test-slave-t6c1"
server_key: "8d72c0b32dc45188e55e7b80661bcfbb"
server_key_iv: "edd2e3026349548f785cecd78922f5c9"
server_key_version: "encryptionkey@0"
I20260929 02:54:29.370210 26501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "1a09090f284d47db82d27a9463fad33e"
format_stamp: "Formatted at 2026-09-29 02:54:29 on dist-test-slave-t6c1"
server_key: "8d72c0b32dc45188e55e7b80661bcfbb"
server_key_iv: "edd2e3026349548f785cecd78922f5c9"
server_key_version: "encryptionkey@0"
I20260929 02:54:29.375527 26501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 02:54:29.379619 26516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:29.381002 26501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:54:29.381171 26501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "1a09090f284d47db82d27a9463fad33e"
format_stamp: "Formatted at 2026-09-29 02:54:29 on dist-test-slave-t6c1"
server_key: "8d72c0b32dc45188e55e7b80661bcfbb"
server_key_iv: "edd2e3026349548f785cecd78922f5c9"
server_key_version: "encryptionkey@0"
I20260929 02:54:29.381374 26501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:29.386986 26501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:29.390765 26501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:29.391080 26501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:29.406996 26501 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43783
I20260929 02:54:29.407047 26567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43783 every 8 connection(s)
I20260929 02:54:29.408382 26501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:29.411747 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 26501
I20260929 02:54:29.411912 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:29.412328 26568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:29.412351 4886 external_mini_cluster.cc:1428] Setting key a758ea9907ee7ba2cf7451aa4c31e591
I20260929 02:54:29.424515 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:29.418224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:57143 (local address 127.0.0.1:43783)
0929 02:54:29.419030 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:29.419046 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:29.419102 (+ 56us) server_negotiation.cc:407] Connection header received
0929 02:54:29.420779 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:29.420807 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:29.421264 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:29.421684 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:29.422627 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:29.422635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:29.422654 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:29.422676 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:29.422745 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:29.422774 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:29.423001 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:29.423133 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 02:54:29.423344 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":299,"threads_started":1}
I20260929 02:54:29.425186 26568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e: Bootstrap starting.
I20260929 02:54:29.428282 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:29.428620 26568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:29.429776 26568 log.cc:824] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:29.432830 26568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e: No bootstrap required, opened a new log
I20260929 02:54:29.435375 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:29.433079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.73:53339 (local address 127.0.0.1:43783)
0929 02:54:29.433233 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:29.433238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:29.433655 (+ 417us) server_negotiation.cc:407] Connection header received
0929 02:54:29.433781 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:29.433786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:29.433840 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:29.433938 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:29.434750 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:29.434756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:29.434761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:29.434781 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:29.434825 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:29.434842 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:29.434995 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:29.435039 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 02:54:29.435097 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 02:54:29.440750 26568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09090f284d47db82d27a9463fad33e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43783 } }
I20260929 02:54:29.441063 26568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:29.441154 26568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a09090f284d47db82d27a9463fad33e, State: Initialized, Role: FOLLOWER
I20260929 02:54:29.441726 26568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09090f284d47db82d27a9463fad33e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43783 } }
I20260929 02:54:29.442001 26568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:29.442130 26568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:29.442276 26568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:29.444314 26568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09090f284d47db82d27a9463fad33e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43783 } }
I20260929 02:54:29.444749 26568 leader_election.cc:302] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a09090f284d47db82d27a9463fad33e; no voters:
I20260929 02:54:29.445240 26568 leader_election.cc:288] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:29.445369 26578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:29.445962 26578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [term 1 LEADER]: Becoming Leader. State: Replica: 1a09090f284d47db82d27a9463fad33e, State: Running, Role: LEADER
I20260929 02:54:29.446481 26578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09090f284d47db82d27a9463fad33e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43783 } }
I20260929 02:54:29.447026 26568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:29.449049 26579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a09090f284d47db82d27a9463fad33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09090f284d47db82d27a9463fad33e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43783 } } }
I20260929 02:54:29.449419 26579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:29.449858 26580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a09090f284d47db82d27a9463fad33e. Latest consensus state: current_term: 1 leader_uuid: "1a09090f284d47db82d27a9463fad33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09090f284d47db82d27a9463fad33e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43783 } } }
I20260929 02:54:29.450124 26580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:29.450279 26585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:29.455796 26585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:29.464861 26585 catalog_manager.cc:1347] Generated new cluster ID: 50ac2dd988c046b6b0cc15397ec81093
I20260929 02:54:29.465008 26585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:29.505810 26585 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:29.506367 26585 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:29.519512 26585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a09090f284d47db82d27a9463fad33e: Generated new TSK 0
I20260929 02:54:29.520567 26585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:29.534153 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26501
W20260929 02:54:29.535272 26563 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.73:57143
W20260929 02:54:29.535953 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26501: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:29.536080 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 26501
2026-09-29T02:54:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1760 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:29 dist-test-slave-t6c1 krb5kdc[26599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:29 dist-test-slave-t6c1 krb5kdc[26599](info): set up 2 sockets
Sep 29 02:54:29 dist-test-slave-t6c1 krb5kdc[26599](info): commencing operation
krb5kdc: starting...
W20260929 02:54:30.535380 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:30 dist-test-slave-t6c1 krb5kdc[26599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:54:30.775712 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34445
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:34445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:31.101357 26612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:31.101661 26612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:31.101801 26612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:31.108990 26612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:31.109114 26612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:31.109191 26612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:31.109272 26612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:31.109328 26612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:31.118993 26612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:34445
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:34445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:31.120208 26612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:31.121441 26612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:31.132669 26617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:31.132591 26620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:31.132629 26618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:31.133369 26612 server_base.cc:1034] running on GCE node
I20260929 02:54:31.134088 26612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:31.135530 26612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:31.136726 26612 hybrid_clock.cc:648] HybridClock initialized: now 1790650471136672 us; error 49 us; skew 500 ppm
I20260929 02:54:31.137054 26612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:31 dist-test-slave-t6c1 krb5kdc[26599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650471, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:31.144320 26612 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:54:31.145818 26612 webserver.cc:460] Webserver started at http://127.4.197.190:45297/ using document root <none> and password file <none>
I20260929 02:54:31.146466 26612 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:31.146579 26612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:31.146858 26612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:31.149780 26612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "1b0f3c40c79f4e04ac91810413d2b985"
format_stamp: "Formatted at 2026-09-29 02:54:31 on dist-test-slave-t6c1"
server_key: "60b05799fc90cf183f3c685b11c274ce"
server_key_iv: "4d8980f3c857b32a3399cce5d4415ace"
server_key_version: "encryptionkey@0"
I20260929 02:54:31.150529 26612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "1b0f3c40c79f4e04ac91810413d2b985"
format_stamp: "Formatted at 2026-09-29 02:54:31 on dist-test-slave-t6c1"
server_key: "60b05799fc90cf183f3c685b11c274ce"
server_key_iv: "4d8980f3c857b32a3399cce5d4415ace"
server_key_version: "encryptionkey@0"
I20260929 02:54:31.155946 26612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:54:31.159437 26627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:31.160765 26612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:54:31.160928 26612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "1b0f3c40c79f4e04ac91810413d2b985"
format_stamp: "Formatted at 2026-09-29 02:54:31 on dist-test-slave-t6c1"
server_key: "60b05799fc90cf183f3c685b11c274ce"
server_key_iv: "4d8980f3c857b32a3399cce5d4415ace"
server_key_version: "encryptionkey@0"
I20260929 02:54:31.161111 26612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:31.184553 26612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:31.189005 26612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:31.189272 26612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:31.206686 26612 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:34445
I20260929 02:54:31.206709 26678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:34445 every 8 connection(s)
I20260929 02:54:31.208205 26612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:31.211931 26679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:31.218379 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 26612
I20260929 02:54:31.218530 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:31.218987 4886 external_mini_cluster.cc:1428] Setting key 4a9a7db3d6bae532151642713be85ee4
I20260929 02:54:31.221776 26679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985: Bootstrap starting.
I20260929 02:54:31.225226 26679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:31.226423 26679 log.cc:824] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:31.229779 26679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985: No bootstrap required, opened a new log
Sep 29 02:54:31 dist-test-slave-t6c1 krb5kdc[26599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:31.239069 26679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b0f3c40c79f4e04ac91810413d2b985" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34445 } }
I20260929 02:54:31.239624 26679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:31.239750 26679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0f3c40c79f4e04ac91810413d2b985, State: Initialized, Role: FOLLOWER
I20260929 02:54:31.240489 26679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b0f3c40c79f4e04ac91810413d2b985" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34445 } }
I20260929 02:54:31.240818 26679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:31.240962 26679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:31.241158 26679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:31.243713 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:31.220774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53549 (local address 127.4.197.190:34445)
0929 02:54:31.221482 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:31.221493 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:31.221558 (+ 65us) server_negotiation.cc:407] Connection header received
0929 02:54:31.223267 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:31.223290 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:31.223701 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:31.224012 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:31.225770 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:31.227382 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:31.229087 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:31.229803 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:31.229985 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:31.235125 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:31.235133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:31.235152 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:31.235199 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:31.238476 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:31.239460 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:31.239467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:31.239475 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:31.239691 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:31.240289 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:31.240294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:31.240297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:31.241073 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:31.241412 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:31.241829 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 02:54:31.242218 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":322,"threads_started":1}
I20260929 02:54:31.244467 26679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b0f3c40c79f4e04ac91810413d2b985" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34445 } }
I20260929 02:54:31.244951 26679 leader_election.cc:302] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b0f3c40c79f4e04ac91810413d2b985; no voters:
I20260929 02:54:31.245550 26679 leader_election.cc:288] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:31.245684 26684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:31.247119 26684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [term 1 LEADER]: Becoming Leader. State: Replica: 1b0f3c40c79f4e04ac91810413d2b985, State: Running, Role: LEADER
I20260929 02:54:31.247663 26679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:31.247718 26684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b0f3c40c79f4e04ac91810413d2b985" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34445 } }
I20260929 02:54:31.248453 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:31.250275 26685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b0f3c40c79f4e04ac91810413d2b985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b0f3c40c79f4e04ac91810413d2b985" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34445 } } }
I20260929 02:54:31.250617 26685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:31.251121 26686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b0f3c40c79f4e04ac91810413d2b985. Latest consensus state: current_term: 1 leader_uuid: "1b0f3c40c79f4e04ac91810413d2b985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b0f3c40c79f4e04ac91810413d2b985" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 34445 } } }
I20260929 02:54:31.251398 26686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:31.259969 26691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:31.265517 26691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:31.277439 26691 catalog_manager.cc:1347] Generated new cluster ID: 806eaf1f04d141cbb0439a9a101d149a
I20260929 02:54:31.277580 26691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:31.296525 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:31.280501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43355 (local address 127.4.197.190:34445)
0929 02:54:31.280638 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:31.280644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:31.281120 (+ 476us) server_negotiation.cc:407] Connection header received
0929 02:54:31.281299 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:31.281305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:31.281387 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:31.281505 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:31.283268 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:31.284462 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:31.286132 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:31.286815 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:31.286982 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:31.289755 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:31.289765 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:31.289770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:31.289811 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:31.293512 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:31.294309 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:31.294315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:31.294318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:31.294487 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:31.294953 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:31.294958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:31.294962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:31.295344 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:31.295611 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:31.295942 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 02:54:31.296129 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 02:54:31.307251 26691 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:31.310611 26691 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:31.327471 26691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b0f3c40c79f4e04ac91810413d2b985: Generated new TSK 0
I20260929 02:54:31.328794 26691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:31.341617 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26612
I20260929 02:54:31.343310 26674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:31.438196 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 26612
W20260929 02:54:31.452412 26597 connection.cc:535] client connection to 127.4.197.190:34445 recv error: Network error: failed to read from TLS socket (remote: 127.4.197.190:34445): Connection reset by peer (error 104)
2026-09-29T02:54:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:31 dist-test-slave-t6c1 krb5kdc[26711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:31 dist-test-slave-t6c1 krb5kdc[26711](info): set up 2 sockets
Sep 29 02:54:31 dist-test-slave-t6c1 krb5kdc[26711](info): commencing operation
krb5kdc: starting...
W20260929 02:54:32.443285 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.011s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:32 dist-test-slave-t6c1 krb5kdc[26711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:54:32.678300 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33909
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:33.003422 26724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:33.003726 26724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:33.003865 26724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:33.011072 26724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:33.011183 26724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:33.011258 26724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:33.011343 26724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:33.011404 26724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:33.020879 26724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:33909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:33.022044 26724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:33.023363 26724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:33.034799 26729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:33.034770 26730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:33.034735 26732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:33.035553 26724 server_base.cc:1034] running on GCE node
I20260929 02:54:33.036377 26724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:33.037848 26724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:33.039021 26724 hybrid_clock.cc:648] HybridClock initialized: now 1790650473038964 us; error 44 us; skew 500 ppm
I20260929 02:54:33.039378 26724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:33 dist-test-slave-t6c1 krb5kdc[26711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650473, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:33.042402 26724 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:54:33.043834 26724 webserver.cc:460] Webserver started at http://127.4.197.190:37219/ using document root <none> and password file <none>
I20260929 02:54:33.044462 26724 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:33.044556 26724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:33.044811 26724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:33.047801 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "0dd261344ad8446b9ef467041a4fed07"
format_stamp: "Formatted at 2026-09-29 02:54:33 on dist-test-slave-t6c1"
server_key: "d25a54e1c2a8feba1d375cedcd6ae854"
server_key_iv: "0b48b61e1960351a078ba2fc325c75fd"
server_key_version: "encryptionkey@0"
I20260929 02:54:33.048507 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "0dd261344ad8446b9ef467041a4fed07"
format_stamp: "Formatted at 2026-09-29 02:54:33 on dist-test-slave-t6c1"
server_key: "d25a54e1c2a8feba1d375cedcd6ae854"
server_key_iv: "0b48b61e1960351a078ba2fc325c75fd"
server_key_version: "encryptionkey@0"
I20260929 02:54:33.053434 26724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:54:33.056666 26739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:33.057920 26724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:54:33.058070 26724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "0dd261344ad8446b9ef467041a4fed07"
format_stamp: "Formatted at 2026-09-29 02:54:33 on dist-test-slave-t6c1"
server_key: "d25a54e1c2a8feba1d375cedcd6ae854"
server_key_iv: "0b48b61e1960351a078ba2fc325c75fd"
server_key_version: "encryptionkey@0"
I20260929 02:54:33.058239 26724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:33.084529 26724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:33.088366 26724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:33.088631 26724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:33.105986 26724 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:33909
I20260929 02:54:33.105994 26790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:33909 every 8 connection(s)
I20260929 02:54:33.107456 26724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:33.110090 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 26724
I20260929 02:54:33.110285 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:33.110849 4886 external_mini_cluster.cc:1428] Setting key f8707ecbe882d490371d76c7e740c27e
I20260929 02:54:33.112226 26791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:33.124604 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:33.112847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41209 (local address 127.4.197.190:33909)
0929 02:54:33.113642 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:33.113656 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:33.113701 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:54:33.115396 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:33.115421 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:33.115918 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:33.116314 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:33.116893 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:33.118255 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:33.120109 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:33.120798 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:33.120927 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:33.122256 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:33.122268 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:33.122283 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:33.122852 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:33.122959 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:33.122997 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:33.123128 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:33.123389 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 02:54:33.123605 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":342,"threads_started":1}
I20260929 02:54:33.125494 26791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07: Bootstrap starting.
I20260929 02:54:33.128160 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:33.128760 26791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:33.129926 26791 log.cc:824] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:33.133015 26791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07: No bootstrap required, opened a new log
I20260929 02:54:33.139164 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:33.132966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52299 (local address 127.4.197.190:33909)
0929 02:54:33.133137 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:33.133143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:33.133536 (+ 393us) server_negotiation.cc:407] Connection header received
0929 02:54:33.133680 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:33.133684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:33.133764 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:33.133873 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:33.134308 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:33.135455 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:33.136689 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:33.137331 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:33.137456 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:33.138415 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:33.138421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:33.138426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:33.138446 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:33.138491 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:33.138511 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:33.138627 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:33.138761 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:54:33.138814 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:54:33.141667 26791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd261344ad8446b9ef467041a4fed07" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33909 } }
I20260929 02:54:33.142024 26791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:33.142099 26791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd261344ad8446b9ef467041a4fed07, State: Initialized, Role: FOLLOWER
I20260929 02:54:33.142614 26791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd261344ad8446b9ef467041a4fed07" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33909 } }
I20260929 02:54:33.142815 26791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:33.142911 26791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:33.143103 26791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:33.145252 26791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd261344ad8446b9ef467041a4fed07" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33909 } }
I20260929 02:54:33.145802 26791 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dd261344ad8446b9ef467041a4fed07; no voters:
I20260929 02:54:33.146354 26791 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:33.146471 26801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:33.147163 26801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [term 1 LEADER]: Becoming Leader. State: Replica: 0dd261344ad8446b9ef467041a4fed07, State: Running, Role: LEADER
I20260929 02:54:33.147665 26801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd261344ad8446b9ef467041a4fed07" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33909 } }
I20260929 02:54:33.148360 26791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:33.150467 26803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd261344ad8446b9ef467041a4fed07. Latest consensus state: current_term: 1 leader_uuid: "0dd261344ad8446b9ef467041a4fed07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd261344ad8446b9ef467041a4fed07" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33909 } } }
I20260929 02:54:33.150779 26803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:33.151096 26802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dd261344ad8446b9ef467041a4fed07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd261344ad8446b9ef467041a4fed07" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33909 } } }
I20260929 02:54:33.151464 26802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:33.153414 26808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:33.159444 26808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:33.169335 26808 catalog_manager.cc:1347] Generated new cluster ID: b22281f821a94cde9c7af8cd8ed89dc5
I20260929 02:54:33.169512 26808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:33.188781 26808 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:33.190696 26808 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:33.205372 26808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dd261344ad8446b9ef467041a4fed07: Generated new TSK 0
I20260929 02:54:33.206215 26808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:33.221479 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26724
W20260929 02:54:33.222566 26786 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41209
W20260929 02:54:33.223304 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:33.223457 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 26724
W20260929 02:54:33.236899 26709 connection.cc:535] client connection to 127.4.197.190:33909 recv error: Network error: failed to read from TLS socket (remote: 127.4.197.190:33909): Connection reset by peer (error 104)
2026-09-29T02:54:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1784 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:33 dist-test-slave-t6c1 krb5kdc[26822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:33 dist-test-slave-t6c1 krb5kdc[26822](info): set up 2 sockets
Sep 29 02:54:33 dist-test-slave-t6c1 krb5kdc[26822](info): commencing operation
krb5kdc: starting...
W20260929 02:54:34.235190 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:34 dist-test-slave-t6c1 krb5kdc[26822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:54:34.474040 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:41531
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:41531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:34.794790 26835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:34.795102 26835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:34.795203 26835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:34.802342 26835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:34.802464 26835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:34.802511 26835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:34.802569 26835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:34.802613 26835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:34.812410 26835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:41531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:41531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:34.813606 26835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:34.814798 26835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:34.825490 26840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:34.825547 26841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:34.825537 26843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:34.826880 26835 server_base.cc:1034] running on GCE node
I20260929 02:54:34.827610 26835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:34.828895 26835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:34.830093 26835 hybrid_clock.cc:648] HybridClock initialized: now 1790650474830052 us; error 36 us; skew 500 ppm
I20260929 02:54:34.830384 26835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:34 dist-test-slave-t6c1 krb5kdc[26822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650474, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:34.836691 26835 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:54:34.838106 26835 webserver.cc:460] Webserver started at http://127.4.197.190:42643/ using document root <none> and password file <none>
I20260929 02:54:34.838744 26835 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:34.838850 26835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:34.839166 26835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:34.842123 26835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "fcf65c0764034571b857d8edc120729c"
format_stamp: "Formatted at 2026-09-29 02:54:34 on dist-test-slave-t6c1"
server_key: "2f395e516537396e1978fef455668809"
server_key_iv: "8bf5ddfcbb67861f8c2841e18135bf13"
server_key_version: "encryptionkey@0"
I20260929 02:54:34.842902 26835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "fcf65c0764034571b857d8edc120729c"
format_stamp: "Formatted at 2026-09-29 02:54:34 on dist-test-slave-t6c1"
server_key: "2f395e516537396e1978fef455668809"
server_key_iv: "8bf5ddfcbb67861f8c2841e18135bf13"
server_key_version: "encryptionkey@0"
I20260929 02:54:34.848402 26835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:54:34.851871 26850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:34.853263 26835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:54:34.853425 26835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "fcf65c0764034571b857d8edc120729c"
format_stamp: "Formatted at 2026-09-29 02:54:34 on dist-test-slave-t6c1"
server_key: "2f395e516537396e1978fef455668809"
server_key_iv: "8bf5ddfcbb67861f8c2841e18135bf13"
server_key_version: "encryptionkey@0"
I20260929 02:54:34.853616 26835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:34.859416 26835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:34.863286 26835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:34.863543 26835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:34.880909 26835 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:41531
I20260929 02:54:34.880940 26901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:41531 every 8 connection(s)
I20260929 02:54:34.882354 26835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:34.886106 26902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:34.887378 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 26835
I20260929 02:54:34.887538 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:34.887962 4886 external_mini_cluster.cc:1428] Setting key 0513747b4f1d13443352d4de7f4ca223
I20260929 02:54:34.895500 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:34.889802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48377 (local address 127.4.197.190:41531)
0929 02:54:34.890576 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:34.890588 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:34.890628 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:54:34.892236 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:34.892260 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:34.892644 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:34.892980 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:34.894013 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:34.894020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:34.894033 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:34.894046 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:34.894104 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:34.894126 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:34.894285 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:34.894381 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 02:54:34.894555 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":343,"threads_started":1}
I20260929 02:54:34.898809 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:34.898804 26902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c: Bootstrap starting.
I20260929 02:54:34.902418 26902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:34.903687 26902 log.cc:824] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:34.905898 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:34.903582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45933 (local address 127.4.197.190:41531)
0929 02:54:34.903750 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:34.903756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:34.904201 (+ 445us) server_negotiation.cc:407] Connection header received
0929 02:54:34.904367 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:34.904371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:34.904430 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:34.904535 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:34.905345 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:34.905349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:34.905352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:34.905366 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:34.905399 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:34.905412 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:34.905490 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:34.905621 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 02:54:34.905660 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:54:34.906862 26902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c: No bootstrap required, opened a new log
I20260929 02:54:34.914049 26902 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf65c0764034571b857d8edc120729c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41531 } }
I20260929 02:54:34.914366 26902 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:34.914454 26902 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf65c0764034571b857d8edc120729c, State: Initialized, Role: FOLLOWER
I20260929 02:54:34.914987 26902 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf65c0764034571b857d8edc120729c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41531 } }
I20260929 02:54:34.915199 26902 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:34.915319 26902 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:34.915462 26902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:34.917467 26902 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf65c0764034571b857d8edc120729c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41531 } }
I20260929 02:54:34.917922 26902 leader_election.cc:302] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcf65c0764034571b857d8edc120729c; no voters:
I20260929 02:54:34.918416 26902 leader_election.cc:288] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:34.918519 26912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:34.919328 26912 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [term 1 LEADER]: Becoming Leader. State: Replica: fcf65c0764034571b857d8edc120729c, State: Running, Role: LEADER
I20260929 02:54:34.919759 26912 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf65c0764034571b857d8edc120729c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41531 } }
I20260929 02:54:34.920225 26902 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:34.922402 26914 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcf65c0764034571b857d8edc120729c. Latest consensus state: current_term: 1 leader_uuid: "fcf65c0764034571b857d8edc120729c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf65c0764034571b857d8edc120729c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41531 } } }
I20260929 02:54:34.922379 26913 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcf65c0764034571b857d8edc120729c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf65c0764034571b857d8edc120729c" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 41531 } } }
I20260929 02:54:34.922817 26914 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:34.922827 26913 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:34.924106 26917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:34.931649 26917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:34.941829 26917 catalog_manager.cc:1347] Generated new cluster ID: 1a9b6ddecee44f8db7fd6f39feb9a47d
I20260929 02:54:34.941949 26917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:34.959853 26917 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:34.960378 26917 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:34.978195 26917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcf65c0764034571b857d8edc120729c: Generated new TSK 0
I20260929 02:54:34.979266 26917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:34.983822 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26835
W20260929 02:54:34.984987 26897 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48377
W20260929 02:54:34.985595 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26835: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:34.985713 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 26835
2026-09-29T02:54:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1763 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:35 dist-test-slave-t6c1 krb5kdc[26933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:35 dist-test-slave-t6c1 krb5kdc[26933](info): set up 2 sockets
Sep 29 02:54:35 dist-test-slave-t6c1 krb5kdc[26933](info): commencing operation
krb5kdc: starting...
W20260929 02:54:35.995255 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:36 dist-test-slave-t6c1 krb5kdc[26933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:54:36.234411 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33499
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:33499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:36.556718 26946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:36.557011 26946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:36.557137 26946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:36.564682 26946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:36.564821 26946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:36.564893 26946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:36.564977 26946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:36.565039 26946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:36.576114 26946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:33499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:33499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:36.577414 26946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:36.578696 26946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:36.589990 26954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:36.590037 26951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:36.590037 26952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:36.590826 26946 server_base.cc:1034] running on GCE node
I20260929 02:54:36.591615 26946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:36.593005 26946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:36.594216 26946 hybrid_clock.cc:648] HybridClock initialized: now 1790650476594166 us; error 46 us; skew 500 ppm
I20260929 02:54:36.594555 26946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:36 dist-test-slave-t6c1 krb5kdc[26933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:36.597572 26946 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:54:36.599030 26946 webserver.cc:460] Webserver started at http://127.4.197.190:32907/ using document root <none> and password file <none>
I20260929 02:54:36.599669 26946 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:36.599772 26946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:36.600023 26946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:36.602998 26946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "ee3061356e7c4616b8f6b309c35a8a7e"
format_stamp: "Formatted at 2026-09-29 02:54:36 on dist-test-slave-t6c1"
server_key: "06e9c3d6ac30c4d9d0de81d20bb89e33"
server_key_iv: "007738224b2f04067949aa2f4842e278"
server_key_version: "encryptionkey@0"
I20260929 02:54:36.603797 26946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "ee3061356e7c4616b8f6b309c35a8a7e"
format_stamp: "Formatted at 2026-09-29 02:54:36 on dist-test-slave-t6c1"
server_key: "06e9c3d6ac30c4d9d0de81d20bb89e33"
server_key_iv: "007738224b2f04067949aa2f4842e278"
server_key_version: "encryptionkey@0"
I20260929 02:54:36.609073 26946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:54:36.612666 26961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:36.614064 26946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:54:36.614214 26946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "ee3061356e7c4616b8f6b309c35a8a7e"
format_stamp: "Formatted at 2026-09-29 02:54:36 on dist-test-slave-t6c1"
server_key: "06e9c3d6ac30c4d9d0de81d20bb89e33"
server_key_iv: "007738224b2f04067949aa2f4842e278"
server_key_version: "encryptionkey@0"
I20260929 02:54:36.614382 26946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:36.635939 26946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:36.639951 26946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:36.640184 26946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:36.657886 26946 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:33499
I20260929 02:54:36.657907 27012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:33499 every 8 connection(s)
I20260929 02:54:36.659299 26946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:36.663235 27013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:36.667281 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 26946
I20260929 02:54:36.667471 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:36.668010 4886 external_mini_cluster.cc:1428] Setting key 2cc3e9fc861aeef3faf4abf82192b419
I20260929 02:54:36.676311 27013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e: Bootstrap starting.
I20260929 02:54:36.680133 27013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:36.681495 27013 log.cc:824] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:54:36 dist-test-slave-t6c1 krb5kdc[26933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:36.684731 27013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e: No bootstrap required, opened a new log
I20260929 02:54:36.690320 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:36.669839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35131 (local address 127.4.197.190:33499)
0929 02:54:36.670556 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:36.670568 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:36.670608 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:54:36.672240 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:36.672263 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:36.672695 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:36.673037 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:36.674725 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:36.676067 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:36.677771 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:36.678458 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:36.678582 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:36.683311 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:36.683318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:36.683333 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:36.683370 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:36.686290 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:36.687128 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:36.687135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:36.687141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:36.687312 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:36.687695 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:36.687698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:36.687700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:36.688239 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:36.688459 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:36.688885 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 02:54:36.689237 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":309,"threads_started":1}
I20260929 02:54:36.694047 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:36.693781 27013 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3061356e7c4616b8f6b309c35a8a7e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33499 } }
I20260929 02:54:36.694168 27013 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:36.694262 27013 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3061356e7c4616b8f6b309c35a8a7e, State: Initialized, Role: FOLLOWER
I20260929 02:54:36.694813 27013 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3061356e7c4616b8f6b309c35a8a7e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33499 } }
I20260929 02:54:36.695065 27013 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:36.695186 27013 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:36.695335 27013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:36.697649 27013 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3061356e7c4616b8f6b309c35a8a7e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33499 } }
I20260929 02:54:36.698210 27013 leader_election.cc:302] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee3061356e7c4616b8f6b309c35a8a7e; no voters:
I20260929 02:54:36.698832 27013 leader_election.cc:288] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:36.698966 27023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:36.700098 27023 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [term 1 LEADER]: Becoming Leader. State: Replica: ee3061356e7c4616b8f6b309c35a8a7e, State: Running, Role: LEADER
I20260929 02:54:36.700615 27023 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3061356e7c4616b8f6b309c35a8a7e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33499 } }
I20260929 02:54:36.701100 27013 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:36.703140 27024 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee3061356e7c4616b8f6b309c35a8a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3061356e7c4616b8f6b309c35a8a7e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33499 } } }
I20260929 02:54:36.703514 27024 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:36.705792 27025 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee3061356e7c4616b8f6b309c35a8a7e. Latest consensus state: current_term: 1 leader_uuid: "ee3061356e7c4616b8f6b309c35a8a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3061356e7c4616b8f6b309c35a8a7e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 33499 } } }
I20260929 02:54:36.706118 27025 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:36.709491 27028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:36.714130 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:36.698190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45507 (local address 127.4.197.190:33499)
0929 02:54:36.698330 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:36.698333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:36.698908 (+ 575us) server_negotiation.cc:407] Connection header received
0929 02:54:36.699109 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:36.699115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:36.699196 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:36.699339 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:36.701231 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:36.702341 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:36.704157 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:36.704823 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:36.704934 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:36.707665 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:36.707671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:36.707674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:36.707701 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:36.710427 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:36.711112 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:36.711118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:36.711121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:36.711266 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:36.711694 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:36.711698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:36.711702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:36.712171 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:36.713373 (+ 1202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:36.713519 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 02:54:36.713764 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:54:36.715416 27028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:36.724974 27028 catalog_manager.cc:1347] Generated new cluster ID: 4951473461af407bbd3cd950b85801f3
I20260929 02:54:36.725073 27028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:36.741053 27028 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:36.742381 27028 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:36.749363 27028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee3061356e7c4616b8f6b309c35a8a7e: Generated new TSK 0
I20260929 02:54:36.750170 27028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:36.760314 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 26946
I20260929 02:54:36.761400 27008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:36.851799 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 26946
2026-09-29T02:54:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1865 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:36 dist-test-slave-t6c1 krb5kdc[27045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:36 dist-test-slave-t6c1 krb5kdc[27045](info): set up 2 sockets
Sep 29 02:54:36 dist-test-slave-t6c1 krb5kdc[27045](info): commencing operation
krb5kdc: starting...
W20260929 02:54:37.875167 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:38 dist-test-slave-t6c1 krb5kdc[27045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:54:38.107638 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:38541
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:42311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:38541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:38.431154 27058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:38.431442 27058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:38.431577 27058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:38.438484 27058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:38.438593 27058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:38.438679 27058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:38.438762 27058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:38.438822 27058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:38.448285 27058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:42311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:38541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:38541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:38.449587 27058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:38.450897 27058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:38.462509 27063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:38.462464 27064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:38.462533 27066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:38.463374 27058 server_base.cc:1034] running on GCE node
I20260929 02:54:38.464100 27058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:38.465449 27058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:38.466642 27058 hybrid_clock.cc:648] HybridClock initialized: now 1790650478466576 us; error 57 us; skew 500 ppm
I20260929 02:54:38.467005 27058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:38 dist-test-slave-t6c1 krb5kdc[27045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:38.469945 27058 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:54:38.471365 27058 webserver.cc:460] Webserver started at http://127.4.197.190:35233/ using document root <none> and password file <none>
I20260929 02:54:38.472002 27058 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:38.472123 27058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:38.472394 27058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:38.475281 27058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "3bca81b507914110a5f2f6fba2ba882e"
format_stamp: "Formatted at 2026-09-29 02:54:38 on dist-test-slave-t6c1"
server_key: "d2d621ef66ce34c92f0a6b60de4ebaa4"
server_key_iv: "c6294b915d48eb216bbd70e4e3db281c"
server_key_version: "encryptionkey@0"
I20260929 02:54:38.475994 27058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "3bca81b507914110a5f2f6fba2ba882e"
format_stamp: "Formatted at 2026-09-29 02:54:38 on dist-test-slave-t6c1"
server_key: "d2d621ef66ce34c92f0a6b60de4ebaa4"
server_key_iv: "c6294b915d48eb216bbd70e4e3db281c"
server_key_version: "encryptionkey@0"
I20260929 02:54:38.480815 27058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:54:38.484064 27073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:38.485296 27058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:54:38.485452 27058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "3bca81b507914110a5f2f6fba2ba882e"
format_stamp: "Formatted at 2026-09-29 02:54:38 on dist-test-slave-t6c1"
server_key: "d2d621ef66ce34c92f0a6b60de4ebaa4"
server_key_iv: "c6294b915d48eb216bbd70e4e3db281c"
server_key_version: "encryptionkey@0"
I20260929 02:54:38.485626 27058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:38.499454 27058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:38.503702 27058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:38.503980 27058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:38.521433 27058 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:38541
I20260929 02:54:38.521445 27124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:38541 every 8 connection(s)
I20260929 02:54:38.522828 27058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:38.526537 27125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:38.528692 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 27058
I20260929 02:54:38.528858 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:38.529289 4886 external_mini_cluster.cc:1428] Setting key f8fc0bc54ce41ee30520414af464908e
I20260929 02:54:38.537803 27125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e: Bootstrap starting.
I20260929 02:54:38.540907 27125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:38.541939 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:38.531058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35127 (local address 127.4.197.190:38541)
0929 02:54:38.531867 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:38.531878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:38.531920 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:54:38.533567 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:38.533589 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:38.534008 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:38.534346 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:38.534968 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:38.536334 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:38.537626 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:38.538345 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:38.538506 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:38.539648 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:38.539656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:38.539670 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:38.540208 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:38.540284 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:38.540310 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:38.540415 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:38.540622 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 02:54:38.540894 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":338,"threads_started":1}
I20260929 02:54:38.541986 27125 log.cc:824] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:38.545241 27125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e: No bootstrap required, opened a new log
I20260929 02:54:38.546025 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:38.553558 27125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bca81b507914110a5f2f6fba2ba882e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38541 } }
I20260929 02:54:38.553994 27125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:38.554121 27125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bca81b507914110a5f2f6fba2ba882e, State: Initialized, Role: FOLLOWER
I20260929 02:54:38.554726 27125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bca81b507914110a5f2f6fba2ba882e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38541 } }
I20260929 02:54:38.554987 27125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:38.555114 27125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:38.555276 27125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:38.556576 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:38.550072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55725 (local address 127.4.197.190:38541)
0929 02:54:38.550260 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:38.550266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:38.550672 (+ 406us) server_negotiation.cc:407] Connection header received
0929 02:54:38.550828 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:38.550833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:38.550911 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:38.551040 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:38.551494 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:38.552611 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:38.553994 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:38.554604 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:38.554722 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:38.555842 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:38.555847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:38.555854 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:38.555874 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:38.555919 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:38.555935 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:38.556015 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:38.556165 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 02:54:38.556270 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:54:38.557854 27125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bca81b507914110a5f2f6fba2ba882e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38541 } }
I20260929 02:54:38.558434 27125 leader_election.cc:302] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bca81b507914110a5f2f6fba2ba882e; no voters:
I20260929 02:54:38.559198 27125 leader_election.cc:288] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:38.559260 27135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:38.560362 27135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [term 1 LEADER]: Becoming Leader. State: Replica: 3bca81b507914110a5f2f6fba2ba882e, State: Running, Role: LEADER
I20260929 02:54:38.560899 27135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bca81b507914110a5f2f6fba2ba882e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38541 } }
I20260929 02:54:38.561728 27125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:38.563490 27137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bca81b507914110a5f2f6fba2ba882e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bca81b507914110a5f2f6fba2ba882e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38541 } } }
I20260929 02:54:38.563825 27137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:38.564458 27136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bca81b507914110a5f2f6fba2ba882e. Latest consensus state: current_term: 1 leader_uuid: "3bca81b507914110a5f2f6fba2ba882e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bca81b507914110a5f2f6fba2ba882e" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 38541 } } }
I20260929 02:54:38.564812 27136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:38.566045 27140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:38.571099 27140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:38.581300 27140 catalog_manager.cc:1347] Generated new cluster ID: 185efcf8996043c99541ee8e1befd873
I20260929 02:54:38.581578 27140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:38.593663 27140 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:38.595588 27140 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:38.604524 27140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bca81b507914110a5f2f6fba2ba882e: Generated new TSK 0
I20260929 02:54:38.605543 27140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:38.611824 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 27058
W20260929 02:54:38.613027 27120 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35127
W20260929 02:54:38.613731 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 27058: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:38.613875 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 27058
2026-09-29T02:54:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1760 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:38 dist-test-slave-t6c1 krb5kdc[27156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:38 dist-test-slave-t6c1 krb5kdc[27156](info): set up 2 sockets
Sep 29 02:54:38 dist-test-slave-t6c1 krb5kdc[27156](info): commencing operation
krb5kdc: starting...
W20260929 02:54:39.623231 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:39 dist-test-slave-t6c1 krb5kdc[27156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:54:39.867933 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:43391
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:43391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:40.189288 27169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:40.189641 27169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:40.189795 27169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:40.197348 27169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:40.197459 27169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:40.197532 27169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:40.197613 27169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:40.197676 27169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:40.207773 27169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:43391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:40.209098 27169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:40.210414 27169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:40.222218 27174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:40.222244 27175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:40.222282 27177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:40.222950 27169 server_base.cc:1034] running on GCE node
I20260929 02:54:40.223834 27169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:40.225381 27169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:40.226598 27169 hybrid_clock.cc:648] HybridClock initialized: now 1790650480226541 us; error 54 us; skew 500 ppm
I20260929 02:54:40.226987 27169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:40 dist-test-slave-t6c1 krb5kdc[27156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650480, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:40.234541 27169 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:54:40.236169 27169 webserver.cc:460] Webserver started at http://127.4.197.190:35265/ using document root <none> and password file <none>
I20260929 02:54:40.236896 27169 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:40.237017 27169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:40.237320 27169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:40.240494 27169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "cabd0a906ba8497d9909f45efc32c6d7"
format_stamp: "Formatted at 2026-09-29 02:54:40 on dist-test-slave-t6c1"
server_key: "2292318693e0a774807c178ba5a94a06"
server_key_iv: "f88fdb687d6c618a89b2828fe471de64"
server_key_version: "encryptionkey@0"
I20260929 02:54:40.241256 27169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "cabd0a906ba8497d9909f45efc32c6d7"
format_stamp: "Formatted at 2026-09-29 02:54:40 on dist-test-slave-t6c1"
server_key: "2292318693e0a774807c178ba5a94a06"
server_key_iv: "f88fdb687d6c618a89b2828fe471de64"
server_key_version: "encryptionkey@0"
I20260929 02:54:40.246582 27169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:54:40.250110 27184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:40.251636 27169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:54:40.251817 27169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "cabd0a906ba8497d9909f45efc32c6d7"
format_stamp: "Formatted at 2026-09-29 02:54:40 on dist-test-slave-t6c1"
server_key: "2292318693e0a774807c178ba5a94a06"
server_key_iv: "f88fdb687d6c618a89b2828fe471de64"
server_key_version: "encryptionkey@0"
I20260929 02:54:40.252014 27169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:40.258014 27169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:40.262069 27169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:40.262353 27169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:40.280124 27169 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:43391
I20260929 02:54:40.280143 27235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:43391 every 8 connection(s)
I20260929 02:54:40.281548 27169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:40.285194 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:40.290766 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 27169
I20260929 02:54:40.290951 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:40.291343 4886 external_mini_cluster.cc:1428] Setting key 08b81bacb9ca8d5eaa563da18f83602c
I20260929 02:54:40.295429 27236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7: Bootstrap starting.
I20260929 02:54:40.298698 27236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:40.298763 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:40.293137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39635 (local address 127.4.197.190:43391)
0929 02:54:40.293952 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:40.293964 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:40.294003 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:54:40.295656 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:40.295678 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:40.296022 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:40.296325 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:40.297360 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:40.297369 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:40.297382 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:40.297399 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:40.297452 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:40.297478 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:40.297629 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:40.297728 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 02:54:40.297903 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":365,"threads_started":1}
I20260929 02:54:40.299978 27236 log.cc:824] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:40.302510 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:40.302881 27236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7: No bootstrap required, opened a new log
I20260929 02:54:40.308880 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:40.306515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40001 (local address 127.4.197.190:43391)
0929 02:54:40.306649 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:40.306652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:40.307110 (+ 458us) server_negotiation.cc:407] Connection header received
0929 02:54:40.307272 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:40.307276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:40.307321 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:40.307407 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:40.308254 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:40.308260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:40.308268 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:54:40.308284 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:40.308325 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:54:40.308339 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:40.308437 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:40.308541 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 02:54:40.308602 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 02:54:40.310577 27236 raft_consensus.cc:348] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd0a906ba8497d9909f45efc32c6d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43391 } }
I20260929 02:54:40.310905 27236 raft_consensus.cc:374] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:40.311043 27236 raft_consensus.cc:729] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabd0a906ba8497d9909f45efc32c6d7, State: Initialized, Role: FOLLOWER
I20260929 02:54:40.311630 27236 consensus_queue.cc:260] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd0a906ba8497d9909f45efc32c6d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43391 } }
I20260929 02:54:40.311866 27236 raft_consensus.cc:388] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:40.312007 27236 raft_consensus.cc:482] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:40.312152 27236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:40.314179 27236 raft_consensus.cc:504] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd0a906ba8497d9909f45efc32c6d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43391 } }
I20260929 02:54:40.314590 27236 leader_election.cc:302] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cabd0a906ba8497d9909f45efc32c6d7; no voters:
I20260929 02:54:40.315079 27236 leader_election.cc:288] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:40.315192 27246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:40.316370 27246 raft_consensus.cc:686] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [term 1 LEADER]: Becoming Leader. State: Replica: cabd0a906ba8497d9909f45efc32c6d7, State: Running, Role: LEADER
I20260929 02:54:40.316819 27246 consensus_queue.cc:237] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd0a906ba8497d9909f45efc32c6d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43391 } }
I20260929 02:54:40.317160 27236 sys_catalog.cc:564] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:40.319583 27247 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cabd0a906ba8497d9909f45efc32c6d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd0a906ba8497d9909f45efc32c6d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43391 } } }
I20260929 02:54:40.319988 27247 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:40.320568 27248 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cabd0a906ba8497d9909f45efc32c6d7. Latest consensus state: current_term: 1 leader_uuid: "cabd0a906ba8497d9909f45efc32c6d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd0a906ba8497d9909f45efc32c6d7" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 43391 } } }
I20260929 02:54:40.320919 27248 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:40.323820 27254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:40.329140 27254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:40.339314 27254 catalog_manager.cc:1347] Generated new cluster ID: 37c678d48ecb47dcb9f0d77aa19a8f71
I20260929 02:54:40.339463 27254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:40.380677 27254 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:40.381099 27254 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:40.390951 27254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cabd0a906ba8497d9909f45efc32c6d7: Generated new TSK 0
I20260929 02:54:40.391713 27254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:40.400772 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 27169
W20260929 02:54:40.401966 27231 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39635
W20260929 02:54:40.402631 4886 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 27169: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:54:40.402771 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 27169
2026-09-29T02:54:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1788 ms)
[----------] 18 tests from AuthTokenIssuingTest (32399 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:40 dist-test-slave-t6c1 krb5kdc[27267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:40 dist-test-slave-t6c1 krb5kdc[27267](info): set up 2 sockets
Sep 29 02:54:40 dist-test-slave-t6c1 krb5kdc[27267](info): commencing operation
krb5kdc: starting...
W20260929 02:54:41.423314 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:41 dist-test-slave-t6c1 krb5kdc[27267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:54:41.667447 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:35301
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:35301,127.4.197.189:39557,127.4.197.188:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:41.986433 27280 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:54:41.986783 27280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:41.986877 27280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:41.987018 27280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:41.994590 27280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:41.994684 27280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:41.994767 27280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:41.994839 27280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:41.994944 27280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:42.005244 27280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:35301,127.4.197.189:39557,127.4.197.188:41185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:35301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:42.006524 27280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:42.007829 27280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:42.018635 27286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:42.018688 27285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:42.018718 27288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:42.019695 27280 server_base.cc:1034] running on GCE node
I20260929 02:54:42.020368 27280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:42.021610 27280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:42.022816 27280 hybrid_clock.cc:648] HybridClock initialized: now 1790650482022756 us; error 34 us; skew 500 ppm
I20260929 02:54:42.023141 27280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:42 dist-test-slave-t6c1 krb5kdc[27267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:42.026079 27280 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:54:42.027490 27280 webserver.cc:460] Webserver started at http://127.4.197.190:41019/ using document root <none> and password file <none>
I20260929 02:54:42.028152 27280 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:42.028249 27280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:42.028541 27280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:42.031468 27280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "3b46ca84a6654aa181eb2022f3430b75"
format_stamp: "Formatted at 2026-09-29 02:54:42 on dist-test-slave-t6c1"
server_key: "28f6064423391ec52d233f257aacb548"
server_key_iv: "ed9cf2b239facd87688b504fad4f0ab4"
server_key_version: "encryptionkey@0"
I20260929 02:54:42.032238 27280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "3b46ca84a6654aa181eb2022f3430b75"
format_stamp: "Formatted at 2026-09-29 02:54:42 on dist-test-slave-t6c1"
server_key: "28f6064423391ec52d233f257aacb548"
server_key_iv: "ed9cf2b239facd87688b504fad4f0ab4"
server_key_version: "encryptionkey@0"
I20260929 02:54:42.037321 27280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:54:42.040699 27295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:42.042084 27280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:54:42.042250 27280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "3b46ca84a6654aa181eb2022f3430b75"
format_stamp: "Formatted at 2026-09-29 02:54:42 on dist-test-slave-t6c1"
server_key: "28f6064423391ec52d233f257aacb548"
server_key_iv: "ed9cf2b239facd87688b504fad4f0ab4"
server_key_version: "encryptionkey@0"
I20260929 02:54:42.042454 27280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:42.061622 27280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:42.065994 27280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:42.066251 27280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:42.083482 27280 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:35301
I20260929 02:54:42.083488 27346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:35301 every 8 connection(s)
I20260929 02:54:42.084873 27280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:42.088090 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 27280
I20260929 02:54:42.088307 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:42.089023 27347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:42.089128 4886 external_mini_cluster.cc:1428] Setting key 02dc2c6e091334ef0709150f50869f62
I20260929 02:54:42.097421 27347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:54:42 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.197.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.189@KRBTEST.COM" created.
I20260929 02:54:42.118495 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.101069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.4.197.190:35301)
0929 02:54:42.101811 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:42.101826 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:42.102237 (+ 411us) server_negotiation.cc:407] Connection header received
0929 02:54:42.104145 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:42.104150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:42.104372 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:42.104542 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:42.105724 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.106890 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.108205 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.108747 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.108914 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:42.112148 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:42.112156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:42.112166 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:42.112199 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:42.114792 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:42.115411 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:42.115416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:42.115419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:42.115551 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:42.115958 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:42.115962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:42.115964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:42.116460 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:42.116617 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:42.117229 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 02:54:42.117463 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":510,"thread_start_us":322,"threads_started":1}
I20260929 02:54:42.118548 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.101071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:35301 (local address 127.0.0.1:57396)
0929 02:54:42.102086 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.102124 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:42.103822 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:42.104572 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:42.104579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:42.105228 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:42.105543 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:42.105560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.107047 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.107051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:42.108082 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:42.108094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.108958 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.108964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:42.109167 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:42.109856 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:42.109877 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:42.111954 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:42.114905 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:42.114930 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:42.114941 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:42.115300 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:42.115647 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:42.115651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:42.115655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:42.115836 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:42.116594 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:42.116602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:42.116857 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 02:54:42.117104 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 02:54:42.117614 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":80,"thread_start_us":640,"threads_started":1}
I20260929 02:54:42.121158 27347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:54:42.122025 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.121694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57340)
0929 02:54:42.121832 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.121928 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:54:42.122243 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
I20260929 02:54:42.123741 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.123384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57350)
0929 02:54:42.123520 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.123590 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:54:42.123940 27347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:39557: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab.
Entry for principal kudu/127.4.197.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab.
I20260929 02:54:42.143504 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.189:39557
--webserver_interface=127.4.197.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab
--principal=kudu/127.4.197.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:35301,127.4.197.189:39557,127.4.197.188:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:54:42.173314 27347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } attempt: 1
I20260929 02:54:42.174482 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.174005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57364)
0929 02:54:42.174211 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.174337 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:54:42.175660 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.175277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57366)
0929 02:54:42.175428 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.175510 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:54:42.175877 27347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:39557: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
I20260929 02:54:42.244136 27347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } attempt: 2
I20260929 02:54:42.245316 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.244828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57380)
0929 02:54:42.245075 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.245175 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 02:54:42.246510 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.246056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57396)
0929 02:54:42.246271 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.246350 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 02:54:42.246699 27347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:39557: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
I20260929 02:54:42.337929 27347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } attempt: 3
I20260929 02:54:42.339097 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.338636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57406)
0929 02:54:42.338833 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.338952 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 02:54:42.340147 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.339725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57408)
0929 02:54:42.339901 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.339982 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 02:54:42.340353 27347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:39557: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
W20260929 02:54:42.465689 27354 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:54:42.466025 27354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:42.466096 27354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:42.466176 27354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:42.473417 27354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:42.473526 27354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:42.473579 27354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:42.473628 27354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:42.473688 27354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:42.483573 27347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } attempt: 4
I20260929 02:54:42.483805 27354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:35301,127.4.197.189:39557,127.4.197.188:41185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab
--principal=kudu/127.4.197.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.189:39557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.197.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:42.484882 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.484316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57412)
0929 02:54:42.484628 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.484740 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260929 02:54:42.485379 27354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:42.485883 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.485520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57428)
0929 02:54:42.485665 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.485756 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 02:54:42.486104 27347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:39557: Network error: Client connection negotiation failed: client connection to 127.4.197.189:39557: connect: Connection refused (error 111)
I20260929 02:54:42.486877 27354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:42.498550 27359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:42.498617 27362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:42.498582 27360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:42.499471 27354 server_base.cc:1034] running on GCE node
I20260929 02:54:42.500283 27354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:42.501683 27354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:42.502898 27354 hybrid_clock.cc:648] HybridClock initialized: now 1790650482502827 us; error 52 us; skew 500 ppm
I20260929 02:54:42.503252 27354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:42 dist-test-slave-t6c1 krb5kdc[27267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:42.506210 27354 init.cc:377] Logged in from keytab as kudu/127.4.197.189@KRBTEST.COM (short username kudu)
I20260929 02:54:42.507704 27354 webserver.cc:460] Webserver started at http://127.4.197.189:41017/ using document root <none> and password file <none>
I20260929 02:54:42.508361 27354 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:42.508448 27354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:42.508718 27354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:42.511646 27354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/data/instance:
uuid: "580abde3f275463ebf4a9f1fa4acb4b7"
format_stamp: "Formatted at 2026-09-29 02:54:42 on dist-test-slave-t6c1"
server_key: "89b64494942ac3c42c21a712c4308109"
server_key_iv: "e334e9354f73f118b4079ca6bea8e952"
server_key_version: "encryptionkey@0"
I20260929 02:54:42.512378 27354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/wal/instance:
uuid: "580abde3f275463ebf4a9f1fa4acb4b7"
format_stamp: "Formatted at 2026-09-29 02:54:42 on dist-test-slave-t6c1"
server_key: "89b64494942ac3c42c21a712c4308109"
server_key_iv: "e334e9354f73f118b4079ca6bea8e952"
server_key_version: "encryptionkey@0"
I20260929 02:54:42.517206 27354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:54:42.520392 27369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:42.521656 27354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 02:54:42.521796 27354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/wal
uuid: "580abde3f275463ebf4a9f1fa4acb4b7"
format_stamp: "Formatted at 2026-09-29 02:54:42 on dist-test-slave-t6c1"
server_key: "89b64494942ac3c42c21a712c4308109"
server_key_iv: "e334e9354f73f118b4079ca6bea8e952"
server_key_version: "encryptionkey@0"
I20260929 02:54:42.521974 27354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:42.538537 27354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:42.542717 27354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:42.543004 27354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:42.560498 27354 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.189:39557
I20260929 02:54:42.560555 27420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.189:39557 every 8 connection(s)
I20260929 02:54:42.561826 27354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
I20260929 02:54:42.565919 27421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:42.567605 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 27354
I20260929 02:54:42.567768 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-1/wal/instance
I20260929 02:54:42.568184 4886 external_mini_cluster.cc:1428] Setting key a39c6ebebe00e9ee060b8d38ee1aab23
I20260929 02:54:42.573230 27421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:54:42 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.197.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.188@KRBTEST.COM" created.
I20260929 02:54:42.594208 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.576129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.4.197.190:35301)
0929 02:54:42.576327 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:42.576333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:42.577600 (+ 1267us) server_negotiation.cc:407] Connection header received
0929 02:54:42.579529 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:42.579536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:42.579631 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:42.579757 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:42.581191 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.582195 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.583713 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.584360 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.584500 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:42.587883 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:42.587891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:42.587897 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:42.587944 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:42.591150 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:42.591852 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:42.591858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:42.591861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:42.592001 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:42.592369 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:42.592372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:42.592374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:42.592691 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:42.592836 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:42.593578 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 02:54:42.593812 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:54:42.595419 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.576615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:35301 (local address 127.0.0.1:57410)
0929 02:54:42.577405 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.577455 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:42.579244 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:42.579930 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:42.579938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:42.580455 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:42.581016 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:42.581034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.582339 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.582343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:42.583574 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:42.583590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.584553 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.584558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:42.584751 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:42.585438 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:42.585460 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:42.587633 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:42.591271 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:42.591278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:42.591290 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:42.591704 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:42.592093 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:42.592096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:42.592099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:42.592259 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:42.592829 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:42.592838 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:42.593176 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 02:54:42.593436 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 02:54:42.593994 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":307,"threads_started":1}
I20260929 02:54:42.597577 27421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:54:42 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
I20260929 02:54:42.611379 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.598210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57432)
0929 02:54:42.598363 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.598379 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:42.598474 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:42.599394 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:42.599399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:42.599822 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:42.599951 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:42.599960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.601293 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.601298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:42.602153 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:42.602167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.602989 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.602995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:42.603193 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:42.603797 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:42.603814 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:42.605866 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:42.608935 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:42.608941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:42.608945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:42.609301 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:42.609676 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:42.609682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:42.609686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:42.609831 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:42.610385 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:42.610396 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:42.610530 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 02:54:42.610657 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 02:54:42.610912 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 02:54:42.611450 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.598437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.4.197.189:39557)
0929 02:54:42.598991 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:42.598998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:42.599025 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:54:42.599072 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:42.599079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:42.599250 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:42.599393 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:42.600115 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.601157 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.602286 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.602834 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.602982 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:42.606061 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:42.606072 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:42.606087 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:42.606117 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:42.608814 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:42.609423 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:42.609430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:42.609432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:42.609546 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:42.609930 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:42.609936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:42.609940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:42.610264 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:42.610458 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:42.610815 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 02:54:42.611053 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":242,"threads_started":1}
I20260929 02:54:42.613888 27421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:54:42.615121 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.614641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53622)
0929 02:54:42.614824 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.614967 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 02:54:42.615373 27370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
I20260929 02:54:42.616745 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.616405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53636)
0929 02:54:42.616542 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.616623 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:54:42.616932 27421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:41185: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab.
Entry for principal kudu/127.4.197.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab.
I20260929 02:54:42.620113 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.188:41185
--webserver_interface=127.4.197.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab
--principal=kudu/127.4.197.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:35301,127.4.197.189:39557,127.4.197.188:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:54:42.666273 27421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } attempt: 1
I20260929 02:54:42.667526 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.667063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53644)
0929 02:54:42.667266 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.667364 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 02:54:42.668705 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.668277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53658)
0929 02:54:42.668488 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.668577 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260929 02:54:42.668989 27421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:41185: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
I20260929 02:54:42.737242 27421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } attempt: 2
I20260929 02:54:42.738472 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.737995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53670)
0929 02:54:42.738211 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.738311 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:54:42.739480 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.739132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53682)
0929 02:54:42.739278 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.739353 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:54:42.739689 27421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:41185: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
I20260929 02:54:42.785391 27347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } attempt: 5
Sep 29 02:54:42 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
I20260929 02:54:42.800413 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.786149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57442)
0929 02:54:42.786376 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.786398 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:42.786531 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:42.786906 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:42.786911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:42.787325 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:42.787507 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:42.787517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.788568 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.788574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:42.789898 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:42.789916 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.790642 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.790648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:42.790883 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:42.791512 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:42.791535 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:42.793953 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:42.797920 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:42.797924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:42.797927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:42.798342 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:42.798808 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:42.798812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:42.798814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:42.798996 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:42.799684 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:42.799687 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:42.799834 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 02:54:42.799940 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 02:54:42.800164 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 02:54:42.800670 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.786291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.4.197.189:39557)
0929 02:54:42.786493 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:42.786498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:42.786519 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:54:42.786669 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:42.786675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:42.786760 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:42.786845 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:42.787673 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.788400 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.790059 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:42.790488 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:42.790584 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:42.794179 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:42.794187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:42.794192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:42.794238 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:42.797714 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:42.798481 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:42.798488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:42.798492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:42.798668 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:42.799103 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:42.799109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:42.799112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:42.799551 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:42.799712 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:42.800087 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 02:54:42.800371 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 02:54:42.801717 27347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:54:42.802668 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.802298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53692)
0929 02:54:42.802459 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.802538 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:54:42.803544 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.803229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53706)
0929 02:54:42.803375 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.803454 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 02:54:42.803722 27347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:41185: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
I20260929 02:54:42.830952 27421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } attempt: 3
I20260929 02:54:42.832170 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.831651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53710)
0929 02:54:42.831880 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.831990 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 02:54:42.833329 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.832876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53720)
0929 02:54:42.833110 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.833200 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260929 02:54:42.833560 27421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:41185: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
I20260929 02:54:42.855005 27347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } attempt: 1
I20260929 02:54:42.856249 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.855736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53732)
0929 02:54:42.855963 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.856066 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 02:54:42.857287 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.856920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53734)
0929 02:54:42.857077 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.857163 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 02:54:42.857511 27347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:41185: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
I20260929 02:54:42.925745 27347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } attempt: 2
I20260929 02:54:42.927151 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.926526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53740)
0929 02:54:42.926797 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.926894 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 02:54:42.928363 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.927945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53752)
0929 02:54:42.928119 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.928222 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:54:42.928606 27347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:41185: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
W20260929 02:54:42.945495 27428 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:54:42.945847 27428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:42.945945 27428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:42.946065 27428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:42.953315 27428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:42.953404 27428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:42.953490 27428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:42.953552 27428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:42.953634 27428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:42.963261 27428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:35301,127.4.197.189:39557,127.4.197.188:41185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab
--principal=kudu/127.4.197.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.188:41185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.197.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:42.964648 27428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:42.966116 27428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 02:54:42.976779 27421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } attempt: 4
W20260929 02:54:42.977828 27434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:42.978106 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.977583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53758)
0929 02:54:42.977837 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.977921 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 02:54:42.978078 27436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:42.978679 27433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:42.979104 27428 server_base.cc:1034] running on GCE node
I20260929 02:54:42.979221 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:42.978851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53768)
0929 02:54:42.979008 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:42.979104 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 02:54:42.979436 27421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:41185: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
I20260929 02:54:42.979910 27428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:42.981271 27428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:42.982481 27428 hybrid_clock.cc:648] HybridClock initialized: now 1790650482982422 us; error 35 us; skew 500 ppm
I20260929 02:54:42.982780 27428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:42 dist-test-slave-t6c1 krb5kdc[27267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:42.985728 27428 init.cc:377] Logged in from keytab as kudu/127.4.197.188@KRBTEST.COM (short username kudu)
I20260929 02:54:42.987100 27428 webserver.cc:460] Webserver started at http://127.4.197.188:40297/ using document root <none> and password file <none>
I20260929 02:54:42.987743 27428 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:42.987838 27428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:42.988121 27428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:42.990962 27428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/data/instance:
uuid: "ef3807dd45104699838b807567c5d15c"
format_stamp: "Formatted at 2026-09-29 02:54:42 on dist-test-slave-t6c1"
server_key: "660e2962741a989805a5bffd09bbb740"
server_key_iv: "3b86ed133d107ae1b09bb53fd681261f"
server_key_version: "encryptionkey@0"
I20260929 02:54:42.991701 27428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/wal/instance:
uuid: "ef3807dd45104699838b807567c5d15c"
format_stamp: "Formatted at 2026-09-29 02:54:42 on dist-test-slave-t6c1"
server_key: "660e2962741a989805a5bffd09bbb740"
server_key_iv: "3b86ed133d107ae1b09bb53fd681261f"
server_key_version: "encryptionkey@0"
I20260929 02:54:42.996546 27428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:54:42.999683 27443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:43.000885 27428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:54:43.001027 27428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/wal
uuid: "ef3807dd45104699838b807567c5d15c"
format_stamp: "Formatted at 2026-09-29 02:54:42 on dist-test-slave-t6c1"
server_key: "660e2962741a989805a5bffd09bbb740"
server_key_iv: "3b86ed133d107ae1b09bb53fd681261f"
server_key_version: "encryptionkey@0"
I20260929 02:54:43.001200 27428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:43.025593 27428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:43.029610 27428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:43.029863 27428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:43.034844 27347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } attempt: 3
I20260929 02:54:43.036068 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.035566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53774)
0929 02:54:43.035812 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:43.035900 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 02:54:43.037160 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.036799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53786)
0929 02:54:43.036986 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:43.037053 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 02:54:43.037353 27347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:41185: Network error: Client connection negotiation failed: client connection to 127.4.197.188:41185: connect: Connection refused (error 111)
I20260929 02:54:43.047087 27428 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.188:41185
I20260929 02:54:43.047120 27494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.188:41185 every 8 connection(s)
I20260929 02:54:43.048422 27428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
I20260929 02:54:43.052163 27495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:43.053323 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 27428
I20260929 02:54:43.053503 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650137986910-4886-0/minicluster-data/master-2/wal/instance
I20260929 02:54:43.054035 4886 external_mini_cluster.cc:1428] Setting key 4c2403485e30b2b22f8f95d723919d6a
I20260929 02:54:43.061309 27495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:54:43 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
Sep 29 02:54:43 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:43.076089 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.056095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.4.197.188:41185)
0929 02:54:43.056888 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.056904 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.056947 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:54:43.058563 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.058585 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.059054 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.059448 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.060977 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.062363 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.063707 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.064756 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.064931 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.068807 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.068816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.068828 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.068868 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.071892 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.072552 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.072558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.072564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.072716 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.073068 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.073072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.073074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.073568 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.073854 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.074284 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.074731 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":322,"threads_started":1}
I20260929 02:54:43.077046 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.063786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:35301 (local address 127.0.0.1:57412)
0929 02:54:43.064455 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:43.064497 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:43.064639 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:43.064996 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:43.065003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:43.065623 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:43.065853 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:43.065861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.066799 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.066802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:43.067817 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:43.067829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.068600 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.068607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:43.068761 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.069443 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:43.069465 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:43.071812 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:43.074602 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:43.074611 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:43.074618 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:43.075128 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:43.075479 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:43.075484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:43.075487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:43.075652 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:43.076131 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:43.076138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:43.076327 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 02:54:43.076503 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 02:54:43.076771 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":281,"threads_started":1}
I20260929 02:54:43.077235 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.063910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.4.197.190:35301)
0929 02:54:43.064121 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.064127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.064586 (+ 459us) server_negotiation.cc:407] Connection header received
0929 02:54:43.064776 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.064780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.064845 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.064924 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.065995 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.066650 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.067964 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.068380 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.068488 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.071993 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.071998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.072001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.072035 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.074447 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.075242 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.075246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.075248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.075376 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.075757 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.075760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.075762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.076014 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.076125 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.076641 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.076882 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:54:43.078675 27495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:54:43 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
Sep 29 02:54:43 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:43.093894 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.079722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.4.197.190:35301)
0929 02:54:43.079898 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.079904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.079919 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:54:43.079962 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.079966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.080029 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.080113 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.081439 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.082242 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.083668 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.084132 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.084242 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.088302 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.088307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.088310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.088343 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.091063 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.091823 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.091830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.091833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.092007 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.092428 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.092433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.092437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.092789 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.093079 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.093350 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.093575 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:54:43.094296 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.079539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.4.197.189:39557)
0929 02:54:43.079756 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.079762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.079780 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:54:43.079837 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.079842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.079916 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.080010 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.080848 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.081576 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.082970 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.083599 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.083719 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.087741 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.087748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.087752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.087794 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.091248 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.091999 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.092006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.092010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.092168 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.092602 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.092607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.092609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.092944 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.093083 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.093609 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.094005 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:54:43.095038 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.079331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:39557 (local address 127.0.0.1:57450)
0929 02:54:43.079501 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:43.079520 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:43.079617 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:43.080053 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:43.080056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:43.080488 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:43.080676 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:43.080687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.081733 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.081736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:43.082787 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:43.082801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.083994 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.084000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:43.084201 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.084833 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:43.084854 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:43.087523 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:43.091392 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:43.091398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:43.091401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:43.091886 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:43.092294 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:43.092300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:43.092304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:43.092482 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:43.093146 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:43.093152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:43.093327 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 02:54:43.094455 (+ 1128us) client_negotiation.cc:240] Negotiation successful
0929 02:54:43.094692 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:54:43.095924 27495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:54:43 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
Sep 29 02:54:43 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
I20260929 02:54:43.108408 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.096670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.4.197.188:41185)
0929 02:54:43.096836 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.096842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.096996 (+ 154us) server_negotiation.cc:407] Connection header received
0929 02:54:43.097079 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.097084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.097149 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.097242 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.097968 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.098809 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.100035 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.100490 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.100609 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.103326 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.103330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.103333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.103373 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.105872 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.106544 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.106547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.106549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.106647 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.107045 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.107050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.107054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.107380 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.107507 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.107903 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.108130 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":57}
I20260929 02:54:43.108457 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.096678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53812)
0929 02:54:43.096917 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:43.096929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:43.097006 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:43.097246 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:43.097248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:43.097662 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:43.097838 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:43.097848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.098970 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.098975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:43.099905 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:43.099918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.100619 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.100623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:43.100806 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.101239 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:43.101254 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:43.103167 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:43.105988 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:43.105993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:43.105997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:43.106446 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:43.106742 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:43.106746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:43.106750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:43.106898 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:43.107495 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:43.107500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:43.107637 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 02:54:43.107786 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 02:54:43.107994 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":4}
I20260929 02:54:43.109524 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.095512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.4.197.189:39557)
0929 02:54:43.095664 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.095669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.095688 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:54:43.095847 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.095851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.095904 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.095972 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.097360 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.098281 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.099638 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.100215 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.100301 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.103996 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.104001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.104004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.104031 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.106763 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.107492 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.107497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.107501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.107639 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.108043 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.108047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.108051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.108395 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.108632 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.109054 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.109264 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:54:43.113901 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:43.113895 27495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c: Bootstrap starting.
I20260929 02:54:43.117537 27495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:43.118542 27495 log.cc:824] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:43.117100 27322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3b46ca84a6654aa181eb2022f3430b75"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57416
I20260929 02:54:43.121112 27495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c: No bootstrap required, opened a new log
I20260929 02:54:43.128135 27495 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } }
I20260929 02:54:43.128473 27495 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3807dd45104699838b807567c5d15c, State: Initialized, Role: FOLLOWER
I20260929 02:54:43.128895 27495 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } }
I20260929 02:54:43.129772 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } } }
I20260929 02:54:43.130160 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:43.130849 27495 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:54:43.140182 27512 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:54:43.140334 27512 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:54:43.137943 27428 master_runner.cc:186] Error getting master registration for 127.4.197.190:35301: OK, instance_id { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" instance_seqno: 1790650482078586 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:54:43.145978 27428 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:54:43.214577 27347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } attempt: 4
Sep 29 02:54:43 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
I20260929 02:54:43.230824 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.215349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53822)
0929 02:54:43.215562 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:43.215582 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:43.215712 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:43.216058 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:43.216063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:43.216563 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:43.216779 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:43.216792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.218204 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.218209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:43.219708 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:43.219725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.220710 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.220717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:43.220936 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.221561 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:43.221584 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:43.224160 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:43.228212 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:43.228216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:43.228219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:43.228571 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:43.228971 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:43.228976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:43.228979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:43.229146 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:43.229819 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:43.229823 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:43.230003 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 02:54:43.230143 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 02:54:43.230434 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:54:43.230863 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.215410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.4.197.188:41185)
0929 02:54:43.215690 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.215696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.215718 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:54:43.215835 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.215840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.215917 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.216018 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.216922 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.218056 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.219848 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.220564 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.220715 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.224357 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.224365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.224370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.224410 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.228080 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.228676 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.228682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.228686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.228850 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.229262 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.229267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.229271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.229699 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.229868 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.230266 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.230496 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260929 02:54:43.236922 27347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75: Bootstrap starting.
I20260929 02:54:43.241147 27347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:43.242501 27347 log.cc:824] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:43.245855 27347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75: No bootstrap required, opened a new log
I20260929 02:54:43.253393 27347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } }
I20260929 02:54:43.253803 27347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b46ca84a6654aa181eb2022f3430b75, State: Initialized, Role: FOLLOWER
I20260929 02:54:43.254305 27347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } }
I20260929 02:54:43.255287 27515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } } }
I20260929 02:54:43.255659 27515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:43.256534 27347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:43.256618 27322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 02:54:43.256814 27322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:43.259474 27322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } }
I20260929 02:54:43.260905 27322 leader_election.cc:288] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [CANDIDATE]: Term 1 election: Requested vote from peers 580abde3f275463ebf4a9f1fa4acb4b7 (127.4.197.189:39557), ef3807dd45104699838b807567c5d15c (127.4.197.188:41185)
I20260929 02:54:43.262527 27470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b46ca84a6654aa181eb2022f3430b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ef3807dd45104699838b807567c5d15c"
I20260929 02:54:43.263064 27470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:43.266561 27470 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b46ca84a6654aa181eb2022f3430b75 in term 1.
I20260929 02:54:43.267215 27296 leader_election.cc:302] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b46ca84a6654aa181eb2022f3430b75, ef3807dd45104699838b807567c5d15c; no voters:
I20260929 02:54:43.267752 27515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:54:43.278645 27421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } attempt: 5
I20260929 02:54:43.280182 27515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [term 1 LEADER]: Becoming Leader. State: Replica: 3b46ca84a6654aa181eb2022f3430b75, State: Running, Role: LEADER
I20260929 02:54:43.281258 27515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } }
W20260929 02:54:43.281982 27449 tablet.cc:2369] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:54:43 dist-test-slave-t6c1 krb5kdc[27267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
I20260929 02:54:43.294144 27523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b46ca84a6654aa181eb2022f3430b75. Latest consensus state: current_term: 1 leader_uuid: "3b46ca84a6654aa181eb2022f3430b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } } }
I20260929 02:54:43.294606 27523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:43.297932 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.279427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:41185 (local address 127.0.0.1:53824)
0929 02:54:43.279872 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:43.279894 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:43.280031 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:43.280957 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:43.280964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:43.281525 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:43.281756 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:43.281771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.283191 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.283198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:43.284730 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:43.284749 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.286022 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.286029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:43.286300 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.287063 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:43.287090 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:43.289992 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:43.294796 (+ 4804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:43.294812 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:43.294816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:43.295481 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:43.296022 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:43.296028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:43.296032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:43.296215 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:43.296863 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:43.296870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:43.297043 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 02:54:43.297182 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 02:54:43.297502 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296}
I20260929 02:54:43.298954 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.280259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.4.197.188:41185)
0929 02:54:43.280450 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.280457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.280477 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:54:43.280544 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.280549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.280631 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.280740 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.281968 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.283018 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.284925 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.285637 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.285795 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.290192 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.290198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.290203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.290249 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.294607 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.295674 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.295681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.295684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.295882 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.296355 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.296360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.296364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.296713 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.298193 (+ 1480us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.298323 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.298551 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:54:43.305001 27421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7: Bootstrap starting.
I20260929 02:54:43.305263 27531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:43.309077 27421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:43.310359 27421 log.cc:824] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:43.316584 27531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260929 02:54:43.319128 27301 tablet.cc:2369] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:54:43.321506 27421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7: No bootstrap required, opened a new log
I20260929 02:54:43.332389 27280 master_runner.cc:186] Error getting master registration for 127.4.197.189:39557: OK, instance_id { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" instance_seqno: 1790650482555515 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:54:43.344306 27280 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:54:43.350419 27470 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [term 1 FOLLOWER]: Refusing update from remote peer 3b46ca84a6654aa181eb2022f3430b75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:54:43.351610 27523 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:54:43.353412 27421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } }
I20260929 02:54:43.353832 27421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580abde3f275463ebf4a9f1fa4acb4b7, State: Initialized, Role: FOLLOWER
I20260929 02:54:43.354365 27421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } }
I20260929 02:54:43.365132 27539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } } }
I20260929 02:54:43.365563 27539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:43.370050 27421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:43.370074 27396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b46ca84a6654aa181eb2022f3430b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "580abde3f275463ebf4a9f1fa4acb4b7"
I20260929 02:54:43.370544 27395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:43.372524 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.329710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.4.197.190:35301)
0929 02:54:43.330195 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.330201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.344623 (+ 14422us) server_negotiation.cc:407] Connection header received
0929 02:54:43.344850 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.344856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.344965 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.345099 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.349039 (+ 3940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.350189 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.353473 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.354209 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.354306 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.360353 (+ 6047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.360362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.360367 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.360418 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.364600 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.366477 (+ 1877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.366482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.366485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.366666 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.368297 (+ 1631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.368303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.368307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.368969 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.369390 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.371853 (+ 2463us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.372125 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334}
I20260929 02:54:43.374348 27395 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [term 1 FOLLOWER]: Refusing update from remote peer 3b46ca84a6654aa181eb2022f3430b75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:54:43.375708 27523 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:54:43.377866 27396 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b46ca84a6654aa181eb2022f3430b75 in term 1.
I20260929 02:54:43.388960 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.330380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.4.197.188:41185)
0929 02:54:43.330582 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.330588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.355883 (+ 25295us) server_negotiation.cc:407] Connection header received
0929 02:54:43.357117 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.357124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.357209 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.357283 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.360985 (+ 3702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.362024 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.368099 (+ 6075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.368768 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.368872 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.381523 (+ 12651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.381530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.381535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.381583 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.385137 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.386454 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.386460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.386463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.386619 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.387126 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.387130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.387134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.387516 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.387764 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.388327 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.388561 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:54:43.393990 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.356467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.4.197.189:39557)
0929 02:54:43.356628 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.356635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.367175 (+ 10540us) server_negotiation.cc:407] Connection header received
0929 02:54:43.367866 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.367872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.367957 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.368032 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.380392 (+ 12360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.381529 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.383454 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.384111 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.384226 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.387374 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:43.387381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:43.387386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:43.387424 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:43.390719 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.391596 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.391603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.391607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.391758 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:43.392204 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:43.392210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:43.392213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:43.392600 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:43.392838 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.393405 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.393634 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 02:54:43.400230 27539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b46ca84a6654aa181eb2022f3430b75. Latest consensus state: current_term: 1 leader_uuid: "3b46ca84a6654aa181eb2022f3430b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } } }
I20260929 02:54:43.400545 27539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:43.402197 27536 mvcc.cc:205] Tried to move back new op lower bound from 7334504379789955072 to 7334504379530788864. Current Snapshot: MvccSnapshot[applied={T|T < 7334504379789955072}]
I20260929 02:54:43.405270 27539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b46ca84a6654aa181eb2022f3430b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } } }
I20260929 02:54:43.405509 27539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:43.408990 27544 mvcc.cc:205] Tried to move back new op lower bound from 7334504379789955072 to 7334504379530788864. Current Snapshot: MvccSnapshot[applied={T|T < 7334504379789955072 or (T in {7334504379789955072})}]
I20260929 02:54:43.411924 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b46ca84a6654aa181eb2022f3430b75. Latest consensus state: current_term: 1 leader_uuid: "3b46ca84a6654aa181eb2022f3430b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } } }
I20260929 02:54:43.412192 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:43.416815 27541 mvcc.cc:205] Tried to move back new op lower bound from 7334504379789955072 to 7334504379530788864. Current Snapshot: MvccSnapshot[applied={T|T < 7334504379789955072}]
I20260929 02:54:43.420075 27523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b46ca84a6654aa181eb2022f3430b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } } }
I20260929 02:54:43.420367 27523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:43.422490 27531 catalog_manager.cc:1347] Generated new cluster ID: cbb90beba1c942e5b2555222521df739
I20260929 02:54:43.422590 27531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:43.428107 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b46ca84a6654aa181eb2022f3430b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } } }
I20260929 02:54:43.428368 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:43.434669 27515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b46ca84a6654aa181eb2022f3430b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b46ca84a6654aa181eb2022f3430b75" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 35301 } } peers { permanent_uuid: "580abde3f275463ebf4a9f1fa4acb4b7" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 39557 } } peers { permanent_uuid: "ef3807dd45104699838b807567c5d15c" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 41185 } } }
I20260929 02:54:43.435034 27515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:43.462306 27531 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:43.464480 27531 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:43.482170 27531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b46ca84a6654aa181eb2022f3430b75: Generated new TSK 0
I20260929 02:54:43.483438 27531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:43.511032 27559 catalog_manager.cc:1259] Loaded cluster ID: cbb90beba1c942e5b2555222521df739
I20260929 02:54:43.523021 27559 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7: loading cluster ID for follower catalog manager: success
I20260929 02:54:43.527431 27559 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7: acquiring CA information for follower catalog manager: success
I20260929 02:54:43.530248 27559 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 580abde3f275463ebf4a9f1fa4acb4b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:54:43.624046 27350 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:43.611440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.4.197.190:35301)
0929 02:54:43.611721 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.611728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.621713 (+ 9985us) server_negotiation.cc:407] Connection header received
0929 02:54:43.622256 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.622262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.622361 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.622449 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.623740 (+ 1291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57436: BlockingRecv error: recv got EOF from 127.0.0.1:57436 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 02:54:43.627921 27498 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:43.612776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.4.197.188:41185)
0929 02:54:43.612970 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.612976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.624823 (+ 11847us) server_negotiation.cc:407] Connection header received
0929 02:54:43.626167 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.626173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.626275 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.626362 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.627649 (+ 1287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53832: BlockingRecv error: recv got EOF from 127.0.0.1:53832 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 02:54:43.633708 27425 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:43.631255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.4.197.189:39557)
0929 02:54:43.631458 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.631464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.632052 (+ 588us) server_negotiation.cc:407] Connection header received
0929 02:54:43.632290 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.632297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.632400 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.632473 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.633463 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57470: BlockingRecv error: recv got EOF from 127.0.0.1:57470 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:54:43.685140 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.665680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.4.197.189:39557)
0929 02:54:43.665890 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.665897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.676714 (+ 10817us) server_negotiation.cc:407] Connection header received
0929 02:54:43.677332 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.677340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.677609 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.677802 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:43.678545 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.679779 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.681816 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.682469 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.682570 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.683427 (+ 857us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:43.684254 (+ 827us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:43.684394 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.684651 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.684775 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 02:54:43.732861 27498 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:43.718596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.4.197.188:41185)
0929 02:54:43.718795 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.718801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.729622 (+ 10821us) server_negotiation.cc:407] Connection header received
0929 02:54:43.730457 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.730463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.730688 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.730799 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.732396 (+ 1597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53844: BlockingRecv error: recv got EOF from 127.0.0.1:53844 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:54:43.790438 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.784777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.4.197.189:39557)
0929 02:54:43.784984 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.784991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.785562 (+ 571us) server_negotiation.cc:407] Connection header received
0929 02:54:43.786055 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.786062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.786149 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.786237 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:43.786678 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.787482 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.788806 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.789270 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.789380 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.789753 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:43.789878 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:43.789950 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.790092 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.790171 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 02:54:43.811794 27498 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:43.804735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.4.197.188:41185)
0929 02:54:43.804941 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.804947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.810379 (+ 5432us) server_negotiation.cc:407] Connection header received
0929 02:54:43.810601 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.810605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.810676 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.810767 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.811561 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53850: BlockingRecv error: recv got EOF from 127.0.0.1:53850 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:54:43.821959 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.816499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.4.197.189:39557)
0929 02:54:43.816679 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.816685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.817138 (+ 453us) server_negotiation.cc:407] Connection header received
0929 02:54:43.817612 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.817616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.817695 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.817777 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:43.818168 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.818901 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.820301 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.820826 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.820934 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.821281 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:43.821381 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:43.821454 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.821583 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.821655 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 02:54:43.832870 27498 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:43.828122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.4.197.188:41185)
0929 02:54:43.828273 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.828278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.831507 (+ 3229us) server_negotiation.cc:407] Connection header received
0929 02:54:43.831722 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.831726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.831799 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.831881 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.832672 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53862: BlockingRecv error: recv got EOF from 127.0.0.1:53862 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 02:54:43.850432 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.844845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.4.197.189:39557)
0929 02:54:43.845057 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.845065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.845652 (+ 587us) server_negotiation.cc:407] Connection header received
0929 02:54:43.845850 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.845854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.845948 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.846034 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:43.846452 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.847414 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.848751 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.849240 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.849353 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.849759 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:43.849874 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:43.849948 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.850094 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.850180 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 02:54:43.859185 27498 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:43.856953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.4.197.188:41185)
0929 02:54:43.857273 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.857280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.857784 (+ 504us) server_negotiation.cc:407] Connection header received
0929 02:54:43.857995 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.858001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.858100 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.858191 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.858958 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53864: BlockingRecv error: recv got EOF from 127.0.0.1:53864 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
I20260929 02:54:43.876091 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.870425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.4.197.189:39557)
0929 02:54:43.870742 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.870749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.871309 (+ 560us) server_negotiation.cc:407] Connection header received
0929 02:54:43.871511 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.871516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.871631 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.871728 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:43.872127 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.872927 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.874258 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.874751 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.874877 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.875311 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:43.875445 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:43.875522 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.875665 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.875756 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 02:54:43.888779 27498 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:43.886720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.4.197.188:41185)
0929 02:54:43.886956 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.886964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.887474 (+ 510us) server_negotiation.cc:407] Connection header received
0929 02:54:43.887643 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.887649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.887728 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.887811 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.888560 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53880: BlockingRecv error: recv got EOF from 127.0.0.1:53880 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:54:43.913168 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.908102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.4.197.189:39557)
0929 02:54:43.908269 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.908275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.908612 (+ 337us) server_negotiation.cc:407] Connection header received
0929 02:54:43.908800 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.908805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.908870 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.908955 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:43.909342 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.910202 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.911597 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.912027 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.912138 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.912536 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:43.912640 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:43.912723 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.912847 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.912931 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 02:54:43.920415 27498 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:43.918592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.4.197.188:41185)
0929 02:54:43.918812 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.918818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.919122 (+ 304us) server_negotiation.cc:407] Connection header received
0929 02:54:43.919269 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.919275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.919368 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.919481 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.920190 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53892: BlockingRecv error: recv got EOF from 127.0.0.1:53892 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:54:43.961748 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:43.955744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.4.197.189:39557)
0929 02:54:43.955981 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.955991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.956222 (+ 231us) server_negotiation.cc:407] Connection header received
0929 02:54:43.956407 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.956414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.956536 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.956693 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:43.957112 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.958315 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.959715 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:43.960396 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:43.960554 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:43.960899 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:43.961035 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:43.961136 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:43.961295 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 02:54:43.961376 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260929 02:54:43.973894 27498 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:43.971939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.4.197.188:41185)
0929 02:54:43.972138 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:43.972145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:43.972463 (+ 318us) server_negotiation.cc:407] Connection header received
0929 02:54:43.972631 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:43.972638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:43.972743 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:43.972888 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:43.973613 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53900: BlockingRecv error: recv got EOF from 127.0.0.1:53900 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:54:44.046473 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:44.041193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.4.197.189:39557)
0929 02:54:44.041373 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:44.041378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:44.041818 (+ 440us) server_negotiation.cc:407] Connection header received
0929 02:54:44.042001 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:44.042006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:44.042075 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:44.042161 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:44.042572 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.043325 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.044680 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.045039 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.045126 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:44.045618 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:44.045799 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:44.045908 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:44.046066 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 02:54:44.046170 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 02:54:44.055025 27498 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:44.053211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.4.197.188:41185)
0929 02:54:44.053381 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:44.053386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:44.053680 (+ 294us) server_negotiation.cc:407] Connection header received
0929 02:54:44.053848 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:44.053852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:44.053920 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:44.053995 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:44.054814 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53910: BlockingRecv error: recv got EOF from 127.0.0.1:53910 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 02:54:44.142443 27512 catalog_manager.cc:1259] Loaded cluster ID: cbb90beba1c942e5b2555222521df739
I20260929 02:54:44.142553 27512 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c: loading cluster ID for follower catalog manager: success
I20260929 02:54:44.145344 27512 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c: acquiring CA information for follower catalog manager: success
I20260929 02:54:44.147529 27512 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ef3807dd45104699838b807567c5d15c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:54:44.190997 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:44.186372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.4.197.189:39557)
0929 02:54:44.186535 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:44.186539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:44.186911 (+ 372us) server_negotiation.cc:407] Connection header received
0929 02:54:44.187067 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:44.187075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:44.187167 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:44.187267 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:44.187671 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.188373 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.189631 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.190019 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.190110 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:44.190418 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:44.190523 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:44.190589 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:44.190678 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 02:54:44.190740 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:54:44.201625 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:44.196286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.4.197.188:41185)
0929 02:54:44.196498 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:44.196504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:44.196673 (+ 169us) server_negotiation.cc:407] Connection header received
0929 02:54:44.196889 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:44.196893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:44.197045 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:44.197144 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:44.197494 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.198172 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.199510 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.199932 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.200039 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:44.200503 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:44.201043 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:44.201184 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:44.201246 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 02:54:44.201328 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:54:44.216208 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:44.208468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.4.197.188:41185)
0929 02:54:44.208660 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:44.208666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:44.209855 (+ 1189us) server_negotiation.cc:407] Connection header received
0929 02:54:44.210202 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:44.210207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:44.210279 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:44.210350 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:44.210795 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.211797 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.214301 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.214821 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.214888 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:44.215603 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:44.215711 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:44.215796 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:44.215917 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 02:54:44.215993 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:54:44.217367 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:44.208332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.4.197.190:35301)
0929 02:54:44.208760 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:44.208766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:44.209720 (+ 954us) server_negotiation.cc:407] Connection header received
0929 02:54:44.210040 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:44.210045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:44.210278 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:44.210448 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:44.211267 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.212183 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.214316 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.214832 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.214975 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:44.215418 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:44.215820 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:44.215914 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:44.217030 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0929 02:54:44.217093 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":213,"threads_started":1}
I20260929 02:54:44.218152 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:44.209942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.4.197.189:39557)
0929 02:54:44.210121 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:44.210126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:44.212809 (+ 2683us) server_negotiation.cc:407] Connection header received
0929 02:54:44.212990 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:44.212996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:44.213062 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:44.213131 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:44.213678 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.214443 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.216281 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:44.216873 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:44.216958 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:44.217422 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:44.217545 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:44.217607 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:44.217836 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 02:54:44.217912 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 02:54:44.221436 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 27280
I20260929 02:54:44.222520 27342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:44.332998 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 27280
I20260929 02:54:44.349781 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 27354
I20260929 02:54:44.350790 27416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:44.456143 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 27354
I20260929 02:54:44.472062 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 27428
I20260929 02:54:44.473153 27490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:44.578312 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 27428
2026-09-29T02:54:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4179 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:44 dist-test-slave-t6c1 krb5kdc[27672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:44 dist-test-slave-t6c1 krb5kdc[27672](info): set up 2 sockets
Sep 29 02:54:44 dist-test-slave-t6c1 krb5kdc[27672](info): commencing operation
krb5kdc: starting...
W20260929 02:54:45.595453 4886 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/test-user.keytab.
Sep 29 02:54:45 dist-test-slave-t6c1 krb5kdc[27672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.197.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
Entry for principal kudu/127.4.197.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab.
I20260929 02:54:45.837739 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45619
--webserver_interface=127.4.197.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:45619,127.4.197.189:41083,127.4.197.188:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:46.161331 27685 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:54:46.161685 27685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:46.161792 27685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:46.161909 27685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:46.169443 27685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:46.169539 27685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:46.169618 27685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:46.169687 27685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:46.169780 27685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:46.179601 27685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:45619,127.4.197.189:41083,127.4.197.188:46799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.190.keytab
--principal=kudu/127.4.197.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.190:45619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.197.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:46.181036 27685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:46.182459 27685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:46.194567 27690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:46.194519 27693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:46.194590 27691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:46.195376 27685 server_base.cc:1034] running on GCE node
I20260929 02:54:46.196085 27685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:46.197444 27685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:46.198637 27685 hybrid_clock.cc:648] HybridClock initialized: now 1790650486198575 us; error 41 us; skew 500 ppm
I20260929 02:54:46.198987 27685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:46 dist-test-slave-t6c1 krb5kdc[27672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:46.205171 27685 init.cc:377] Logged in from keytab as kudu/127.4.197.190@KRBTEST.COM (short username kudu)
I20260929 02:54:46.206511 27685 webserver.cc:460] Webserver started at http://127.4.197.190:37727/ using document root <none> and password file <none>
I20260929 02:54:46.207235 27685 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:46.207324 27685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:46.207578 27685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:46.210424 27685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/data/instance:
uuid: "95c882e640f04b39baaa5f912eee5bab"
format_stamp: "Formatted at 2026-09-29 02:54:46 on dist-test-slave-t6c1"
server_key: "32f9a29d061c90538c009354b2e9cd9d"
server_key_iv: "a4b365084321746dce3f4f85ea15f079"
server_key_version: "encryptionkey@0"
I20260929 02:54:46.211184 27685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/wal/instance:
uuid: "95c882e640f04b39baaa5f912eee5bab"
format_stamp: "Formatted at 2026-09-29 02:54:46 on dist-test-slave-t6c1"
server_key: "32f9a29d061c90538c009354b2e9cd9d"
server_key_iv: "a4b365084321746dce3f4f85ea15f079"
server_key_version: "encryptionkey@0"
I20260929 02:54:46.215936 27685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:54:46.219013 27700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:46.220259 27685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:54:46.220408 27685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/wal
uuid: "95c882e640f04b39baaa5f912eee5bab"
format_stamp: "Formatted at 2026-09-29 02:54:46 on dist-test-slave-t6c1"
server_key: "32f9a29d061c90538c009354b2e9cd9d"
server_key_iv: "a4b365084321746dce3f4f85ea15f079"
server_key_version: "encryptionkey@0"
I20260929 02:54:46.220585 27685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:46.237732 27685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:46.241940 27685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:46.242198 27685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:46.259197 27685 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.190:45619
I20260929 02:54:46.259218 27751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.190:45619 every 8 connection(s)
I20260929 02:54:46.260592 27685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:46.261898 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 27685
I20260929 02:54:46.262074 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-0/wal/instance
I20260929 02:54:46.262575 4886 external_mini_cluster.cc:1428] Setting key 18d388b72c36ba79a62ab97e98c3e7b7
I20260929 02:54:46.265328 27752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:46.273833 27752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.197.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.189@KRBTEST.COM" created.
Sep 29 02:54:46 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:46.299347 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.277207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45619 (local address 127.0.0.1:46406)
0929 02:54:46.279494 (+ 2287us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.279541 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:46.281429 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:46.282449 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:46.282454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:46.283111 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:46.283485 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:46.283503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.285750 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.285754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:46.286888 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:46.286900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.288219 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.288225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:46.288345 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:46.288973 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:46.288996 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:46.292249 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:46.295305 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:46.295317 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:46.295337 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:46.295694 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:46.296067 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:46.296072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:46.296076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:46.296229 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:46.296960 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:46.296971 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:46.297179 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 02:54:46.297399 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 02:54:46.297989 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2008,"mutex_wait_us":124,"spinlock_wait_cycles":56576,"thread_start_us":1776,"threads_started":1}
I20260929 02:54:46.299367 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.277207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.4.197.190:45619)
0929 02:54:46.278111 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:46.278125 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:46.279689 (+ 1564us) server_negotiation.cc:407] Connection header received
0929 02:54:46.281836 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:46.281843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:46.282185 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:46.282424 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:46.283699 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.285580 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.287055 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.287967 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.288155 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:46.292495 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:46.292503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:46.292517 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:46.292548 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:46.295186 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:46.295818 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:46.295825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:46.295834 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:46.295969 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:46.296333 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:46.296337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:46.296339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:46.296837 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:46.297020 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:46.297557 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 02:54:46.297839 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":448,"threads_started":1}
I20260929 02:54:46.302441 27752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:54:46.303560 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.303127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33002)
0929 02:54:46.303296 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.303430 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 02:54:46.303833 27702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
I20260929 02:54:46.305222 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.304873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33006)
0929 02:54:46.305012 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.305085 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:54:46.305406 27752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:41083: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab.
Entry for principal kudu/127.4.197.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab.
I20260929 02:54:46.318450 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.189:41083
--webserver_interface=127.4.197.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab
--principal=kudu/127.4.197.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:45619,127.4.197.189:41083,127.4.197.188:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:54:46.354756 27752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } attempt: 1
I20260929 02:54:46.356145 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.355607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33018)
0929 02:54:46.355837 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.355980 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 02:54:46.357182 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.356769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33026)
0929 02:54:46.356927 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.357013 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:54:46.357363 27752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:41083: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
I20260929 02:54:46.425613 27752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } attempt: 2
I20260929 02:54:46.426895 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.426319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33042)
0929 02:54:46.426592 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.426713 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:54:46.427969 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.427551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33054)
0929 02:54:46.427719 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.427816 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:54:46.428170 27752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:41083: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
I20260929 02:54:46.519445 27752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } attempt: 3
I20260929 02:54:46.520705 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.520202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33064)
0929 02:54:46.520433 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.520544 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 02:54:46.521780 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.521341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33072)
0929 02:54:46.521512 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.521603 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 02:54:46.521981 27752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:41083: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
W20260929 02:54:46.642266 27759 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:54:46.642643 27759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:46.642745 27759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:46.642861 27759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:46.650126 27759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:46.650218 27759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:46.650288 27759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:46.650358 27759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:46.650436 27759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:46.660185 27759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:45619,127.4.197.189:41083,127.4.197.188:46799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.189.keytab
--principal=kudu/127.4.197.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.189:41083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.197.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:46.661437 27759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:46.662701 27759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 02:54:46.665197 27752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } attempt: 4
I20260929 02:54:46.666353 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.665747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33076)
0929 02:54:46.666060 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.666170 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 02:54:46.667408 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.667005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33090)
0929 02:54:46.667149 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.667250 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 02:54:46.667599 27752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.189:41083: Network error: Client connection negotiation failed: client connection to 127.4.197.189:41083: connect: Connection refused (error 111)
W20260929 02:54:46.674099 27764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:46.674067 27767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:46.674147 27765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:46.675163 27759 server_base.cc:1034] running on GCE node
I20260929 02:54:46.675872 27759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:46.677169 27759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:46.678406 27759 hybrid_clock.cc:648] HybridClock initialized: now 1790650486678336 us; error 41 us; skew 500 ppm
I20260929 02:54:46.678717 27759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:46 dist-test-slave-t6c1 krb5kdc[27672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:46.681699 27759 init.cc:377] Logged in from keytab as kudu/127.4.197.189@KRBTEST.COM (short username kudu)
I20260929 02:54:46.683139 27759 webserver.cc:460] Webserver started at http://127.4.197.189:36201/ using document root <none> and password file <none>
I20260929 02:54:46.683817 27759 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:46.683909 27759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:46.684202 27759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:46.687192 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/data/instance:
uuid: "98bd34e9cc1d4321831dc57cd24e4eb4"
format_stamp: "Formatted at 2026-09-29 02:54:46 on dist-test-slave-t6c1"
server_key: "0f75989ec929f0de560e1821c261f6ff"
server_key_iv: "c6ba41a266f64aba927e493b63d96b5f"
server_key_version: "encryptionkey@0"
I20260929 02:54:46.687937 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/wal/instance:
uuid: "98bd34e9cc1d4321831dc57cd24e4eb4"
format_stamp: "Formatted at 2026-09-29 02:54:46 on dist-test-slave-t6c1"
server_key: "0f75989ec929f0de560e1821c261f6ff"
server_key_iv: "c6ba41a266f64aba927e493b63d96b5f"
server_key_version: "encryptionkey@0"
I20260929 02:54:46.692837 27759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:54:46.695909 27774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:46.697149 27759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 02:54:46.697293 27759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/wal
uuid: "98bd34e9cc1d4321831dc57cd24e4eb4"
format_stamp: "Formatted at 2026-09-29 02:54:46 on dist-test-slave-t6c1"
server_key: "0f75989ec929f0de560e1821c261f6ff"
server_key_iv: "c6ba41a266f64aba927e493b63d96b5f"
server_key_version: "encryptionkey@0"
I20260929 02:54:46.697474 27759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:46.720863 27759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:46.725390 27759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:46.725653 27759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:46.742316 27759 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.189:41083
I20260929 02:54:46.742333 27825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.189:41083 every 8 connection(s)
I20260929 02:54:46.743695 27759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/data/info.pb
I20260929 02:54:46.747273 27826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:46.751529 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 27759
I20260929 02:54:46.751705 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-1/wal/instance
I20260929 02:54:46.752274 4886 external_mini_cluster.cc:1428] Setting key 255fb2b4e303daf47c24320be84bdcd5
I20260929 02:54:46.754961 27826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:54:46 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:46.778553 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.758132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.4.197.190:45619)
0929 02:54:46.758371 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:46.758378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:46.759799 (+ 1421us) server_negotiation.cc:407] Connection header received
0929 02:54:46.761781 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:46.761787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:46.761885 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:46.762013 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:46.764207 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.764946 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.767101 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.767556 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.767664 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:46.772133 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:46.772147 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:46.772153 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:46.772199 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:46.775057 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:46.775956 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:46.775960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:46.775963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:46.776102 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:46.776558 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:46.776562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:46.776564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:46.776878 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:46.777029 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:46.777951 (+ 922us) server_negotiation.cc:299] Negotiation successful
0929 02:54:46.778219 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.4.197.188@KRBTEST.COM; defaulting to no policy
I20260929 02:54:46.779659 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.758698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45619 (local address 127.0.0.1:46410)
0929 02:54:46.759600 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.759651 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:46.761491 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:46.762318 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:46.762331 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:46.763181 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:46.763986 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:46.764014 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.765113 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.765118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:46.766889 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:46.766904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.767792 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.767799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:46.768108 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:46.768890 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:46.768920 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:46.771699 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:46.775216 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:46.775227 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:46.775245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:46.775808 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:46.776214 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:46.776219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:46.776221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:46.776434 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:46.777011 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:46.777020 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:46.777532 (+ 512us) client_negotiation.cc:769] Sending connection context
0929 02:54:46.777797 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 02:54:46.778394 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":361,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.197.188@KRBTEST.COM" created.
I20260929 02:54:46.781479 27826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:54:46 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
I20260929 02:54:46.795802 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.782364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33094)
0929 02:54:46.782566 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.782585 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:46.782699 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:46.783789 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:46.783793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:46.784156 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:46.784296 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:46.784305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.785952 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.785956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:46.786767 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:46.786777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.787794 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.787798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:46.787929 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:46.788495 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:46.788510 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:46.790582 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:46.793441 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:46.793446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:46.793451 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:46.793790 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:46.794126 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:46.794131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:46.794134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:46.794257 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:46.794813 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:46.794818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:46.794967 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 02:54:46.795093 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 02:54:46.795374 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 02:54:46.795872 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.782545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.4.197.189:41083)
0929 02:54:46.783275 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:46.783283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:46.783314 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:54:46.783390 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:46.783399 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:46.783634 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:46.783779 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:46.784464 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.785820 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.786902 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.787657 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.787778 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:46.790767 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:46.790777 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:46.790791 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:46.790831 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:46.793322 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:46.793906 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:46.793912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:46.793914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:46.794020 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:46.794355 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:46.794360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:46.794363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:46.794683 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:46.794872 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:46.795242 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 02:54:46.795492 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":566,"thread_start_us":415,"threads_started":1}
I20260929 02:54:46.798051 27826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:54:46.799132 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.798661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33374)
0929 02:54:46.798845 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.798995 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:54:46.799376 27777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
I20260929 02:54:46.800880 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.800575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33388)
0929 02:54:46.800701 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.800776 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:54:46.801123 27826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:46799: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.197.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab.
Entry for principal kudu/127.4.197.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab.
I20260929 02:54:46.809935 4886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
/tmp/dist-test-taskgLoZGx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.188:46799
--webserver_interface=127.4.197.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab
--principal=kudu/127.4.197.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.197.148:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.197.190:45619,127.4.197.189:41083,127.4.197.188:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:54:46.850484 27826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } attempt: 1
I20260929 02:54:46.851835 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.851271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33402)
0929 02:54:46.851500 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.851629 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:54:46.852952 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.852524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33412)
0929 02:54:46.852710 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.852814 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 02:54:46.853195 27826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:46799: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
I20260929 02:54:46.921468 27826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } attempt: 2
I20260929 02:54:46.922734 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.922199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33416)
0929 02:54:46.922421 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.922531 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 02:54:46.923870 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.923477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33424)
0929 02:54:46.923629 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.923728 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:54:46.924147 27826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:46799: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
I20260929 02:54:46.966908 27752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } attempt: 5
Sep 29 02:54:46 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
I20260929 02:54:46.980813 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.967826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.4.197.189:41083)
0929 02:54:46.968063 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:46.968069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:46.968091 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:54:46.968229 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:46.968234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:46.968305 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:46.968386 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:46.969210 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.969925 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.971213 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.971913 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.972055 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:46.975337 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:46.975342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:46.975346 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:46.975382 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:46.978049 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:46.978949 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:46.978961 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:46.978965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:46.979120 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:46.979501 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:46.979504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:46.979506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:46.979794 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:46.979905 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:46.980387 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 02:54:46.980559 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 02:54:46.980908 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.967729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33096)
0929 02:54:46.967918 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.967941 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:46.968090 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:46.968440 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:46.968445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:46.968848 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:46.969059 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:46.969069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.970058 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.970063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:46.971040 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:46.971057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:46.972072 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:46.972077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:46.972253 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:46.972885 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:46.972902 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:46.975139 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:46.978197 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:46.978203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:46.978208 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:46.978777 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:46.979240 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:46.979244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:46.979248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:46.979411 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:46.979906 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:46.979911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:46.980096 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 02:54:46.980273 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 02:54:46.980565 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 02:54:46.981889 27752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:54:46.982901 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.982508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33428)
0929 02:54:46.982656 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.982762 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 02:54:46.983835 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:46.983526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33440)
0929 02:54:46.983648 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:46.983740 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 02:54:46.984019 27752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:46799: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
I20260929 02:54:47.015390 27826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } attempt: 3
I20260929 02:54:47.016546 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.016015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33452)
0929 02:54:47.016240 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.016353 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 02:54:47.017700 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.017235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33454)
0929 02:54:47.017459 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.017554 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260929 02:54:47.017927 27826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:46799: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
I20260929 02:54:47.035228 27752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } attempt: 1
I20260929 02:54:47.036444 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.035962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33466)
0929 02:54:47.036177 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.036279 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 02:54:47.037442 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.037093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33478)
0929 02:54:47.037262 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.037336 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 02:54:47.037654 27752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:46799: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
I20260929 02:54:47.105854 27752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } attempt: 2
I20260929 02:54:47.107029 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.106521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33492)
0929 02:54:47.106739 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.106828 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 02:54:47.107968 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.107659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33508)
0929 02:54:47.107794 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.107861 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:54:47.108165 27752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:46799: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
W20260929 02:54:47.128654 27833 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:54:47.128988 27833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:47.129078 27833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:47.129199 27833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:47.136442 27833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:47.136534 27833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:47.136598 27833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:47.136669 27833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:47.136750 27833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:47.146638 27833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.197.148:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.197.190:45619,127.4.197.189:41083,127.4.197.188:46799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/krb5kdc/kudu_127.4.197.188.keytab
--principal=kudu/127.4.197.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.197.188:46799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.197.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:47.147975 27833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:47.149322 27833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:47.160579 27839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:47.160714 27838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:47.160580 27841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:47.161162 27826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } attempt: 4
I20260929 02:54:47.161494 27833 server_base.cc:1034] running on GCE node
I20260929 02:54:47.162315 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.161787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33520)
0929 02:54:47.162022 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.162124 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 02:54:47.162424 27833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:47.163401 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.163048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33530)
0929 02:54:47.163182 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.163264 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 02:54:47.163621 27826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:46799: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
I20260929 02:54:47.164260 27833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:47.165532 27833 hybrid_clock.cc:648] HybridClock initialized: now 1790650487165451 us; error 36 us; skew 500 ppm
I20260929 02:54:47.165938 27833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:47 dist-test-slave-t6c1 krb5kdc[27672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650487, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:47.169317 27833 init.cc:377] Logged in from keytab as kudu/127.4.197.188@KRBTEST.COM (short username kudu)
I20260929 02:54:47.170845 27833 webserver.cc:460] Webserver started at http://127.4.197.188:41737/ using document root <none> and password file <none>
I20260929 02:54:47.171583 27833 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:47.171681 27833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:47.171980 27833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:47.175087 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/data/instance:
uuid: "48a3bc13ec6c4d6c9d9a79f027362c74"
format_stamp: "Formatted at 2026-09-29 02:54:47 on dist-test-slave-t6c1"
server_key: "837a91ec0ba57d84c492c09622770d9b"
server_key_iv: "6ba6743d4fb98346572d413663cac2ae"
server_key_version: "encryptionkey@0"
I20260929 02:54:47.175884 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/wal/instance:
uuid: "48a3bc13ec6c4d6c9d9a79f027362c74"
format_stamp: "Formatted at 2026-09-29 02:54:47 on dist-test-slave-t6c1"
server_key: "837a91ec0ba57d84c492c09622770d9b"
server_key_iv: "6ba6743d4fb98346572d413663cac2ae"
server_key_version: "encryptionkey@0"
I20260929 02:54:47.181238 27833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:54:47.184814 27848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:47.186170 27833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:54:47.186334 27833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/data,/tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/wal
uuid: "48a3bc13ec6c4d6c9d9a79f027362c74"
format_stamp: "Formatted at 2026-09-29 02:54:47 on dist-test-slave-t6c1"
server_key: "837a91ec0ba57d84c492c09622770d9b"
server_key_iv: "6ba6743d4fb98346572d413663cac2ae"
server_key_version: "encryptionkey@0"
I20260929 02:54:47.186535 27833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:47.199404 27833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:47.203577 27833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:47.203822 27833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:47.214360 27752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } attempt: 3
I20260929 02:54:47.215487 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.215012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33534)
0929 02:54:47.215191 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.215291 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 02:54:47.216459 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.216110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33544)
0929 02:54:47.216285 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.216358 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 02:54:47.216642 27752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.197.188:46799: Network error: Client connection negotiation failed: client connection to 127.4.197.188:46799: connect: Connection refused (error 111)
I20260929 02:54:47.220736 27833 rpc_server.cc:307] RPC server started. Bound to: 127.4.197.188:46799
I20260929 02:54:47.220753 27899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.197.188:46799 every 8 connection(s)
I20260929 02:54:47.221998 27833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/data/info.pb
I20260929 02:54:47.225689 27900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:47.231997 4886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu as pid 27833
I20260929 02:54:47.232164 4886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgLoZGx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650137986910-4886-0/minicluster-data/master-2/wal/instance
I20260929 02:54:47.232573 4886 external_mini_cluster.cc:1428] Setting key a950bbc6218f57aeeeb8eabc085d27b1
I20260929 02:54:47.232336 27900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:54:47 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650487, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
Sep 29 02:54:47 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
I20260929 02:54:47.251406 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.235125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.4.197.190:45619)
0929 02:54:47.235315 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.235320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.236085 (+ 765us) server_negotiation.cc:407] Connection header received
0929 02:54:47.237427 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.237431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.237497 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.237578 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.238997 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.239717 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.241423 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.242000 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.242095 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.245824 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.245829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.245833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.245867 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.248405 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.249187 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.249193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.249196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.249320 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.249703 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.249708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.249711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.249971 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.250110 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.250797 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.251046 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:54:47.252506 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.234315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.4.197.188:46799)
0929 02:54:47.235116 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.235127 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.235169 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:54:47.237159 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.237194 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.237511 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.237702 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.239043 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.240436 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.241773 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.242512 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.242638 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.246252 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.246260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.246273 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.246301 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.249116 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.249758 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.249764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.249766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.249887 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.250232 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.250236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.250238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.250574 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.250864 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.251159 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.251398 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":539,"thread_start_us":320,"threads_started":1}
I20260929 02:54:47.252542 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.235178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.190:45619 (local address 127.0.0.1:46418)
0929 02:54:47.235915 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.235968 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:47.237174 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:47.237669 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:47.237675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:47.238437 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:47.238802 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:47.238821 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.239870 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.239875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:47.241275 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:47.241291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.242241 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.242249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:47.242539 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.243302 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:47.243334 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:47.245632 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:47.248530 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:47.248542 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:47.248556 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:47.249077 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:47.249417 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:47.249422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:47.249425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:47.249599 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:47.250098 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:47.250106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:47.250446 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 02:54:47.250685 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 02:54:47.251237 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":1,"thread_start_us":312,"threads_started":1}
I20260929 02:54:47.254086 27900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:54:47 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650487, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
Sep 29 02:54:47 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.190@KRBTEST.COM
I20260929 02:54:47.268651 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.255038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.189:41083 (local address 127.0.0.1:33102)
0929 02:54:47.255226 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.255240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:47.255346 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:47.255757 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:47.255761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:47.256096 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:47.256269 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:47.256280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.257598 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.257603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:47.258991 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:47.259003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.259641 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.259646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:47.259834 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.260355 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:47.260378 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:47.262636 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:47.266072 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:47.266077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:47.266081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:47.266593 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:47.267038 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:47.267041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:47.267044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:47.267198 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:47.267692 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:47.267697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:47.267865 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 02:54:47.268004 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 02:54:47.268302 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:54:47.268688 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.254898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.4.197.189:41083)
0929 02:54:47.255103 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.255109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.255312 (+ 203us) server_negotiation.cc:407] Connection header received
0929 02:54:47.255544 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.255550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.255633 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.255737 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.256424 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.257449 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.259112 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.259512 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.259620 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.262914 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.262944 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.262950 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.262991 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.265942 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.266718 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.266723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.266726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.266886 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.267303 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.267307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.267311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.267576 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.267961 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.268099 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.268302 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:54:47.269923 27900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:54:47.272567 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.255613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.4.197.190:45619)
0929 02:54:47.255968 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.255972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.255993 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:54:47.256032 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.256036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.256089 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.256187 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.257599 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.258757 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.260430 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.261042 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.261158 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.266065 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.266072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.266077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.266117 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.269772 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.270504 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.270509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.270513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.270665 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.271122 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.271127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.271131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.271479 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.271681 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.272030 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.272255 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
Sep 29 02:54:47 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650487, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.188@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
Sep 29 02:54:47 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.197.189@KRBTEST.COM
I20260929 02:54:47.282912 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.270607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33568)
0929 02:54:47.270786 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.270804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:47.270905 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:47.271242 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:47.271245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:47.271603 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:47.271779 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:47.271791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.273020 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.273025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:47.274137 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:47.274153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.275020 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.275027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:47.275244 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.275762 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:47.275784 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:47.277907 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:47.280821 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:47.280826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:47.280830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:47.281169 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:47.281503 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:47.281508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:47.281512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:47.281635 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:47.282145 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:47.282148 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:47.282277 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 02:54:47.282415 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 02:54:47.282627 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 02:54:47.283169 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.270648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.4.197.188:46799)
0929 02:54:47.270798 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.270803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.270845 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:54:47.271046 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.271049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.271116 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.271208 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.271923 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.272892 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.274286 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.274710 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.274812 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.278068 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.278072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.278075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.278103 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.280690 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.281273 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.281279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.281282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.281403 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.281727 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.281732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.281736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.282026 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.282171 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.282543 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.282754 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:54:47.288357 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.274259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.4.197.189:41083)
0929 02:54:47.274434 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.274440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.274511 (+ 71us) server_negotiation.cc:407] Connection header received
0929 02:54:47.274653 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.274658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.274727 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.274808 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.276266 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.277060 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.278370 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.278949 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.279074 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.283165 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.283172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.283176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.283218 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.285837 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.286618 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.286622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.286624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.286758 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.287164 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.287167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.287169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.287430 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.287630 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.287945 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.288117 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:54:47.288771 27900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74: Bootstrap starting.
I20260929 02:54:47.292328 27900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:47.292821 4886 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:54:47.293390 27900 log.cc:824] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:47.296442 27900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74: No bootstrap required, opened a new log
I20260929 02:54:47.295408 27727 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "95c882e640f04b39baaa5f912eee5bab"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46426
I20260929 02:54:47.306721 27900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } }
I20260929 02:54:47.307150 27900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a3bc13ec6c4d6c9d9a79f027362c74, State: Initialized, Role: FOLLOWER
I20260929 02:54:47.307651 27900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } }
I20260929 02:54:47.308482 27906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } } }
I20260929 02:54:47.308772 27906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:47.309653 27900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:54:47.318822 27917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:54:47.318986 27917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:54:47.316413 27833 master_runner.cc:186] Error getting master registration for 127.4.197.190:45619: OK, instance_id { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" instance_seqno: 1790650486254361 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:54:47.324898 27833 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:54:47.393852 27752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } attempt: 4
Sep 29 02:54:47 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.190@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
I20260929 02:54:47.411738 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.394644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33576)
0929 02:54:47.395131 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.395153 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:47.395281 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:47.395865 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:47.395870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:47.396400 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:47.396616 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:47.396630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.398086 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.398092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:47.399651 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:47.399670 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.400700 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.400707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:47.400941 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.401647 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:47.401674 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:47.404544 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:47.408752 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:47.408758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:47.408763 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:47.409325 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:47.409777 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:47.409783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:47.409787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:47.409974 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:47.410791 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:47.410796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:47.410990 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 02:54:47.411111 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 02:54:47.411413 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327}
I20260929 02:54:47.412099 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.395152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.4.197.188:46799)
0929 02:54:47.395535 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.395542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.395559 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:54:47.395619 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.395625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.395694 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.395780 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.396783 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.397926 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.399801 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.400536 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.400672 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.404769 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.404777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.404782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.404828 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.408604 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.409461 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.409465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.409468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.409652 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.410102 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.410108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.410111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.410635 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.410821 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.411268 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.411528 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260929 02:54:47.417788 27752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab: Bootstrap starting.
I20260929 02:54:47.421610 27752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:47.422830 27752 log.cc:824] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:47.426491 27752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab: No bootstrap required, opened a new log
I20260929 02:54:47.434387 27752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } }
I20260929 02:54:47.434846 27752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c882e640f04b39baaa5f912eee5bab, State: Initialized, Role: FOLLOWER
I20260929 02:54:47.435509 27752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } }
I20260929 02:54:47.436681 27920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } } }
I20260929 02:54:47.437093 27920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:47.438081 27752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:47.438158 27727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 02:54:47.438335 27727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:47.442194 27727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } }
I20260929 02:54:47.445832 27727 leader_election.cc:288] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [CANDIDATE]: Term 1 election: Requested vote from peers 98bd34e9cc1d4321831dc57cd24e4eb4 (127.4.197.189:41083), 48a3bc13ec6c4d6c9d9a79f027362c74 (127.4.197.188:46799)
I20260929 02:54:47.446691 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95c882e640f04b39baaa5f912eee5bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74"
I20260929 02:54:47.447265 27875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:47.451592 27875 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95c882e640f04b39baaa5f912eee5bab in term 1.
W20260929 02:54:47.454655 27931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:54:47.454769 27931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:54:47.455228 27703 leader_election.cc:302] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48a3bc13ec6c4d6c9d9a79f027362c74, 95c882e640f04b39baaa5f912eee5bab; no voters:
I20260929 02:54:47.455694 27920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [term 1 FOLLOWER]: Leader election won for term 1
W20260929 02:54:47.455772 27854 tablet.cc:2369] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:54:47.462821 27826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } attempt: 5
Sep 29 02:54:47 dist-test-slave-t6c1 krb5kdc[27672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.197.189@KRBTEST.COM for kudu/127.4.197.188@KRBTEST.COM
I20260929 02:54:47.482842 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.463645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.197.188:46799 (local address 127.0.0.1:33580)
0929 02:54:47.464091 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:47.464115 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:47.464250 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:47.465195 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:47.465201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:47.465766 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:47.466011 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:47.466025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.467630 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.467637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:47.469074 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:47.469092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.470667 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.470673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:47.470938 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.471724 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:47.471751 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:47.474950 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:47.479788 (+ 4838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:47.479794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:47.479798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:47.480389 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:47.480868 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:47.480873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:47.480876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:47.481039 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:47.481742 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:47.481747 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:47.481918 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 02:54:47.482064 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 02:54:47.482372 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260929 02:54:47.484115 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.464482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.4.197.188:46799)
0929 02:54:47.464672 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.464678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.464698 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:54:47.464763 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.464768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.464875 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.464979 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.466232 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.467456 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.469261 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.469980 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.470105 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.475220 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.475228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.475234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.475281 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.479574 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.480537 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.480542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.480545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.480718 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.481166 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.481171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.481174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.481537 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.481680 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.483304 (+ 1624us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.483713 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:54:47.490487 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4: Bootstrap starting.
W20260929 02:54:47.495088 27706 tablet.cc:2369] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:54:47.450321 27685 master_runner.cc:186] Error getting master registration for 127.4.197.189:41083: OK, instance_id { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" instance_seqno: 1790650486737507 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:54:47.495491 27685 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:54:47.495702 27826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:47.497152 27826 log.cc:824] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:47.500912 27920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [term 1 LEADER]: Becoming Leader. State: Replica: 95c882e640f04b39baaa5f912eee5bab, State: Running, Role: LEADER
I20260929 02:54:47.501583 27920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } }
I20260929 02:54:47.503063 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4: No bootstrap required, opened a new log
I20260929 02:54:47.516024 27826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } }
I20260929 02:54:47.516710 27826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bd34e9cc1d4321831dc57cd24e4eb4, State: Initialized, Role: FOLLOWER
I20260929 02:54:47.517328 27826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } }
I20260929 02:54:47.520247 27934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95c882e640f04b39baaa5f912eee5bab. Latest consensus state: current_term: 1 leader_uuid: "95c882e640f04b39baaa5f912eee5bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } } }
I20260929 02:54:47.520649 27934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:47.537971 27943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:47.543905 27943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:47.541577 27940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } } }
I20260929 02:54:47.544530 27940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:47.545867 27826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:47.545902 27801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95c882e640f04b39baaa5f912eee5bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4"
I20260929 02:54:47.550024 27801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:47.553682 27801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95c882e640f04b39baaa5f912eee5bab in term 1.
I20260929 02:54:47.568595 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.520292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.4.197.189:41083)
0929 02:54:47.521091 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.521098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.527345 (+ 6247us) server_negotiation.cc:407] Connection header received
0929 02:54:47.527619 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.527625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.527709 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.527781 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.543055 (+ 15274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.545359 (+ 2304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.547405 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.550708 (+ 3303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.550854 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.556570 (+ 5716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.556578 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.556583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.556622 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.560191 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.560994 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.560999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.561003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.561184 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.564659 (+ 3475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.564665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.564669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.565073 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.565326 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.567930 (+ 2604us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.568221 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671}
I20260929 02:54:47.585004 27875 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [term 1 FOLLOWER]: Refusing update from remote peer 95c882e640f04b39baaa5f912eee5bab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:54:47.586165 27934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:54:47.590178 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.527320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.4.197.188:46799)
0929 02:54:47.527516 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.527522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.543884 (+ 16362us) server_negotiation.cc:407] Connection header received
0929 02:54:47.544133 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.544139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.544249 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.544320 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.548851 (+ 4531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.549874 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.565831 (+ 15957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.566589 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.566695 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.577894 (+ 11199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.577902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.577908 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.577956 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.581598 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.582568 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.582574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.582577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.582742 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.588122 (+ 5380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.588129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.588134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.588560 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.588834 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.589472 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.589737 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:54:47.594152 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.521361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.4.197.190:45619)
0929 02:54:47.521536 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.521542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.538778 (+ 17236us) server_negotiation.cc:407] Connection header received
0929 02:54:47.541977 (+ 3199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.541983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.542070 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.542147 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.561652 (+ 19505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.562733 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.573803 (+ 11070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.574559 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.574671 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.583344 (+ 8673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:47.583351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:47.583357 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:47.583423 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:47.587163 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.591565 (+ 4402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.591572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.591576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.591762 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:47.592316 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:47.592321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:47.592325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:47.592716 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:47.592988 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.593515 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.593779 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:54:47.604058 27906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95c882e640f04b39baaa5f912eee5bab. Latest consensus state: current_term: 1 leader_uuid: "95c882e640f04b39baaa5f912eee5bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } } }
I20260929 02:54:47.604394 27906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:47.606317 27801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [term 1 FOLLOWER]: Refusing update from remote peer 95c882e640f04b39baaa5f912eee5bab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:54:47.607473 27934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:54:47.618640 27948 mvcc.cc:205] Tried to move back new op lower bound from 7334504397129031680 to 7334504396812185600. Current Snapshot: MvccSnapshot[applied={T|T < 7334504397129031680}]
I20260929 02:54:47.623940 27946 mvcc.cc:205] Tried to move back new op lower bound from 7334504397129031680 to 7334504396812185600. Current Snapshot: MvccSnapshot[applied={T|T < 7334504397129031680}]
I20260929 02:54:47.634388 27934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95c882e640f04b39baaa5f912eee5bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } } }
I20260929 02:54:47.634701 27934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:47.635594 27943 catalog_manager.cc:1347] Generated new cluster ID: 89a93c72225e413caa5acddbec05225f
I20260929 02:54:47.635687 27943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:47.636883 27906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95c882e640f04b39baaa5f912eee5bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } } }
I20260929 02:54:47.637110 27906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:47.658097 27940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95c882e640f04b39baaa5f912eee5bab. Latest consensus state: current_term: 1 leader_uuid: "95c882e640f04b39baaa5f912eee5bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } } }
I20260929 02:54:47.658444 27940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:47.658061 27955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95c882e640f04b39baaa5f912eee5bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } } }
I20260929 02:54:47.658934 27955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:54:47.663429 27934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95c882e640f04b39baaa5f912eee5bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c882e640f04b39baaa5f912eee5bab" member_type: VOTER last_known_addr { host: "127.4.197.190" port: 45619 } } peers { permanent_uuid: "98bd34e9cc1d4321831dc57cd24e4eb4" member_type: VOTER last_known_addr { host: "127.4.197.189" port: 41083 } } peers { permanent_uuid: "48a3bc13ec6c4d6c9d9a79f027362c74" member_type: VOTER last_known_addr { host: "127.4.197.188" port: 46799 } } }
I20260929 02:54:47.663693 27934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:47.667506 27943 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:47.669528 27943 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:47.697083 27943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95c882e640f04b39baaa5f912eee5bab: Generated new TSK 0
I20260929 02:54:47.698292 27943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:54:47.838078 27967 catalog_manager.cc:1259] Loaded cluster ID: 89a93c72225e413caa5acddbec05225f
I20260929 02:54:47.838302 27967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4: loading cluster ID for follower catalog manager: success
I20260929 02:54:47.842317 27967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4: acquiring CA information for follower catalog manager: success
I20260929 02:54:47.845017 27967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98bd34e9cc1d4321831dc57cd24e4eb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:54:47.856220 27755 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:47.847169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.4.197.190:45619)
0929 02:54:47.847375 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.847380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.851640 (+ 4260us) server_negotiation.cc:407] Connection header received
0929 02:54:47.851866 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.851872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.851990 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.852094 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.855940 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39526: BlockingRecv error: recv got EOF from 127.0.0.1:39526 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 02:54:47.857689 27830 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:47.847013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.4.197.189:41083)
0929 02:54:47.847196 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.847202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.852336 (+ 5134us) server_negotiation.cc:407] Connection header received
0929 02:54:47.854317 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.854324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.854409 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.854480 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.857425 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54916: BlockingRecv error: recv got EOF from 127.0.0.1:54916 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 02:54:47.860199 27903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:47.852549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.4.197.188:46799)
0929 02:54:47.852719 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.852725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.858475 (+ 5750us) server_negotiation.cc:407] Connection header received
0929 02:54:47.858636 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.858642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.858734 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.858838 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.859980 (+ 1142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58914: BlockingRecv error: recv got EOF from 127.0.0.1:58914 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:54:47.919173 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:47.900459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.4.197.189:41083)
0929 02:54:47.900635 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.900642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.911443 (+ 10801us) server_negotiation.cc:407] Connection header received
0929 02:54:47.911803 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.911810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.912039 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.912193 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:47.912952 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.914177 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.916261 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:47.916918 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:47.917014 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:47.917721 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:47.918370 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:47.918492 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:47.918720 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 02:54:47.918817 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 02:54:47.969697 27903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:47.959555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.4.197.188:46799)
0929 02:54:47.959755 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:47.959761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:47.967359 (+ 7598us) server_negotiation.cc:407] Connection header received
0929 02:54:47.967954 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:47.967962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:47.968097 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:47.968176 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:47.969438 (+ 1262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58926: BlockingRecv error: recv got EOF from 127.0.0.1:58926 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:54:48.021028 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.003298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.4.197.189:41083)
0929 02:54:48.003530 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.003536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.014000 (+ 10464us) server_negotiation.cc:407] Connection header received
0929 02:54:48.014345 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.014352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.014455 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.014540 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.015328 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.016510 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.018669 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.019373 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.019473 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.020117 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.020305 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.020364 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.020598 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.020700 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 02:54:48.069324 27903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:48.055059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.4.197.188:46799)
0929 02:54:48.055281 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.055288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.067584 (+ 12296us) server_negotiation.cc:407] Connection header received
0929 02:54:48.067784 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.067790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.067896 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.067993 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:48.069094 (+ 1101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58934: BlockingRecv error: recv got EOF from 127.0.0.1:58934 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:54:48.128587 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.110663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.4.197.189:41083)
0929 02:54:48.110851 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.110857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.121607 (+ 10750us) server_negotiation.cc:407] Connection header received
0929 02:54:48.122065 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.122071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.122154 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.122216 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.122945 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.124236 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.126250 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.126964 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.127078 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.127692 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.127853 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.127920 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.128180 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.128266 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 02:54:48.143707 27903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:48.141688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.4.197.188:46799)
0929 02:54:48.141883 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.141890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.142350 (+ 460us) server_negotiation.cc:407] Connection header received
0929 02:54:48.142527 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.142533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.142611 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.142694 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:48.143489 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58942: BlockingRecv error: recv got EOF from 127.0.0.1:58942 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:54:48.155761 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.150583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.4.197.189:41083)
0929 02:54:48.150810 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.150815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.151019 (+ 204us) server_negotiation.cc:407] Connection header received
0929 02:54:48.151177 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.151183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.151280 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.151380 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.151665 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.152739 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.154014 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.154701 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.154806 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.155168 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.155283 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.155365 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.155431 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.155502 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260929 02:54:48.162389 27903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:48.160782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.4.197.188:46799)
0929 02:54:48.160986 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.160992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.161228 (+ 236us) server_negotiation.cc:407] Connection header received
0929 02:54:48.161356 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.161361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.161440 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.161534 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:48.162178 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58946: BlockingRecv error: recv got EOF from 127.0.0.1:58946 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 02:54:48.179028 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.173276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.4.197.189:41083)
0929 02:54:48.173471 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.173477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.173774 (+ 297us) server_negotiation.cc:407] Connection header received
0929 02:54:48.173977 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.173982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.174068 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.174163 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.174546 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.175691 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.177047 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.177712 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.177848 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.178221 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.178382 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.178466 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.178588 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.178677 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 02:54:48.187722 27903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:48.185857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.4.197.188:46799)
0929 02:54:48.186048 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.186054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.186360 (+ 306us) server_negotiation.cc:407] Connection header received
0929 02:54:48.186532 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.186536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.186603 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.186679 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:48.187485 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58954: BlockingRecv error: recv got EOF from 127.0.0.1:58954 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:54:48.212363 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.206778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54960 (local address 127.4.197.189:41083)
0929 02:54:48.206977 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.206983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.207322 (+ 339us) server_negotiation.cc:407] Connection header received
0929 02:54:48.207473 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.207479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.207569 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.207663 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.208082 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.209254 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.210533 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.211192 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.211318 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.211667 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.211794 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.211885 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.212009 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.212091 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 02:54:48.220024 27903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:48.218130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.4.197.188:46799)
0929 02:54:48.218307 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.218312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.218588 (+ 276us) server_negotiation.cc:407] Connection header received
0929 02:54:48.218766 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.218770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.218839 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.218952 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:48.219758 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58970: BlockingRecv error: recv got EOF from 127.0.0.1:58970 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:54:48.262804 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.255945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.4.197.189:41083)
0929 02:54:48.256152 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.256159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.256703 (+ 544us) server_negotiation.cc:407] Connection header received
0929 02:54:48.257089 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.257096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.257188 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.257263 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.257931 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.258743 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.260672 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.261413 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.261534 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.262011 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.262151 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.262248 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.262404 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.262495 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 02:54:48.274544 27903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:54:48.272718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58978 (local address 127.4.197.188:46799)
0929 02:54:48.272882 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.272887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.272913 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:54:48.273219 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.273225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.273314 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.273422 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:48.274306 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58978: BlockingRecv error: recv got EOF from 127.0.0.1:58978 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:54:48.320847 27917 catalog_manager.cc:1259] Loaded cluster ID: 89a93c72225e413caa5acddbec05225f
I20260929 02:54:48.320943 27917 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74: loading cluster ID for follower catalog manager: success
I20260929 02:54:48.323447 27917 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74: acquiring CA information for follower catalog manager: success
I20260929 02:54:48.325376 27917 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 48a3bc13ec6c4d6c9d9a79f027362c74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:54:48.348448 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.343326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.4.197.189:41083)
0929 02:54:48.343566 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.343572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.343821 (+ 249us) server_negotiation.cc:407] Connection header received
0929 02:54:48.344050 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.344055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.344125 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.344201 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.344634 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.345369 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.346737 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.347209 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.347331 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.347741 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.347867 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.347940 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.348091 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.348174 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 02:54:48.359115 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.353713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.4.197.188:46799)
0929 02:54:48.353915 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.353919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.354264 (+ 345us) server_negotiation.cc:407] Connection header received
0929 02:54:48.354419 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.354423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.354594 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.354693 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.355034 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.355757 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.357173 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.357572 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.357661 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.358121 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.358648 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.358754 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.358795 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.358865 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:54:48.371373 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.364807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.4.197.189:41083)
0929 02:54:48.365027 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.365033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.365309 (+ 276us) server_negotiation.cc:407] Connection header received
0929 02:54:48.365541 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.365547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.365641 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.365731 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.366553 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.367565 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.369254 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.369932 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.370048 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.370595 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.370751 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.370848 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.371001 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.371086 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 02:54:48.372450 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.364751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.4.197.190:45619)
0929 02:54:48.365344 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.365350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.366000 (+ 650us) server_negotiation.cc:407] Connection header received
0929 02:54:48.366128 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.366134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.366377 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.366539 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.367290 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.368289 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.370013 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.370618 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.370718 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.371399 (+ 681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.371863 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.371950 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.372066 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.372151 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":378,"threads_started":1}
I20260929 02:54:48.372646 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:48.365884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.4.197.188:46799)
0929 02:54:48.366080 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:48.366086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:48.366656 (+ 570us) server_negotiation.cc:407] Connection header received
0929 02:54:48.366789 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:48.366794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:48.366885 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:48.367009 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:54:48.367565 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.368556 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.370478 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:48.371075 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:48.371198 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:48.371770 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:54:48.371898 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:54:48.371980 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:48.372205 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 02:54:48.372283 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:54:48.377174 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 27685
I20260929 02:54:48.378116 27747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:48.488252 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 27685
I20260929 02:54:48.505177 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 27759
I20260929 02:54:48.506172 27821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:48.610503 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 27759
I20260929 02:54:48.626689 4886 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu pid 27833
I20260929 02:54:48.627957 27895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:54:48.736086 4886 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgLoZGx/build/asan/bin/kudu with pid 27833
2026-09-29T02:54:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4157 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8337 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (350713 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST